Bug 11411 - check-uniq-files target issue
Summary: check-uniq-files target issue
Status: RESOLVED FIXED
Alias: None
Product: buildroot
Classification: Unclassified
Component: Other (show other bugs)
Version: 2018.02.4
Hardware: All Linux
: P5 normal
Target Milestone: ---
Assignee: Yann E. MORIN
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-10-19 15:37 UTC by Jean-pierre Cartal
Modified: 2019-10-31 22:07 UTC (History)
1 user (show)

See Also:
Host:
Target:
Build:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jean-pierre Cartal 2018-10-19 15:37:39 UTC
Hello,

The check-uniq-files script is returning lots of false positive when successive builds are launched.

Those detected in the target directory seems to be coming from automatic stripping that is done in the final step in the target directory:
STRIP_FIND_CMD = find $(TARGET_DIR)
ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS)))
STRIP_FIND_CMD += \( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o
endif
STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)

When run this commmand modify the modification date of the stripped files, so on next build, the check-uniq-files script will detect all modified files as being touched by all packages e.g.:

Warning: target file "./usr/lib/libdevmapper-event-lvm2thin.so" is touched by more than one package: [u'lvm2', u'popt', u'util-linux', u'cryptsetup', u'expat', u'dbus', u'e2fsprogs', u'kmod', u'eudev', u'libpng', u'freetype', u'libopenssl', u'ffmpeg', u'giflib', u'icu', u'libffi', u'pcre', u'libglib2', u'harfbuzz', u'hdparm', u'ifupdown-scripts', u'initscripts', u'iperf', u'iw', u'jpeg-turbo', u'jpeg', u'json-glib', u'libcap', u'openssl', u'libcurl', u'libevent', u'libexif', u'libnspr', u'sqlite', u'libnss', u'liburiparser', u'libxml2', u'monit', u'openssh', u'udev', u'pciutils', u'strace', u'sysvinit', u'wireless-regdb', u'wpa_supplicant', u'netgem_bluez5_utils', u'brcm-patchram', u'netgem-azure-c-shared-utility', u'netgem-azure-uamqp-c', u'netgem-azure-umqtt-c', u'netgem-azure-iot-sdk-c', u'netgem_boot-bin', u'netgem_boot', u'netgem_erpc', u'cobalt', u'netgem-bootloader', u'netgem_nexus_standby', u'netgem_secure_dma_code', u'realtek_8822BE_BT_driver', u'busybox']

Maybe the -p strip option could be used to avoid this issue ?

Regards.
Comment 1 Arnout Vandecappelle 2018-10-20 12:17:04 UTC
Sounds like an excellent idea! The -p option has existed since 2000 so no problem there. Care to send a patch to add it?
Comment 2 Jean-pierre Cartal 2018-10-20 16:33:33 UTC
Here is the patch that I tested on my builds:

diff --git a/package/Makefile.in b/package/Makefile.in
index 58af2ef242..7d3d7c9028 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -216,7 +216,7 @@ TARGET_OBJDUMP  = $(TARGET_CROSS)objdump
 ifeq ($(BR2_STRIP_strip),y)
 STRIP_STRIP_DEBUG := --strip-debug
 TARGET_STRIP = $(TARGET_CROSS)strip
-STRIPCMD = $(TARGET_CROSS)strip --remove-section=.comment --remove-section=.note
+STRIPCMD = $(TARGET_CROSS)strip --remove-section=.comment --remove-section=.note -p
 else
 TARGET_STRIP = /bin/true
 STRIPCMD = $(TARGET_STRIP)


BTW, there are also a lot of false positive in staging because of this command :
$(Q)find $(STAGING_DIR)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
                $(SED) "s:$(BASE_DIR):@BASE_DIR@:g" \
                        -e "s:$(STAGING_DIR):@STAGING_DIR@:g" \
                        $(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
                                -e "s:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:g") \
                        -e "s:\(['= ]\)/usr:\\1@STAGING_DIR@/usr:g" \
                        $(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
                                -e "s:@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):g") \
                        -e "s:@STAGING_DIR@:$(STAGING_DIR):g" \
                        -e "s:@BASE_DIR@:$(BASE_DIR):g"

Do you think a patch using stat/touch to keep the original modification time of impacted files be interesting ?
Comment 3 Arnout Vandecappelle 2018-10-20 18:46:36 UTC
The originally reported problem actually only occurs when you do a package-reinstall. Not really the most important use case I'd say. If you do a package rebuild, it works correctly.

For the *.la files, the problem indeed occurs, but it already occurs on the first run. Indeed, after every package, we fix up the .la files, and even if they are not really modified, their timestamp is updated. So that really has to be fixed. However, a better fix is to not update all the .la files, but only the .la files really touched by that package.
Comment 4 Jean-pierre Cartal 2018-10-20 18:59:01 UTC
Regarding the original issue, it was not my impression that the bug was only happening on package reinstall.

For instance the example I gave with 
Warning: target file "./usr/lib/libdevmapper-event-lvm2thin.so" was not triggered by a reinstallation of the lvm package if that's what you mean ?
Comment 5 Arnout Vandecappelle 2018-10-20 22:16:16 UTC
As I said, I couldn't reproduce it, so how did you trigger it?
Comment 6 Jean-pierre Cartal 2018-10-22 07:37:09 UTC
You're correct, this is happening when I force a package reinstallation, a full package rebuild does not trigger the issue.

However, I think that adding the -p option would still be useful to avoid this issue when reinstalling a package.

BTW I created a separate ticket for warnings in staging area (#11416)
Comment 7 Jean-pierre Cartal 2019-10-09 15:51:22 UTC
Hi,

Is there any hope for this patch to be added to main buildroot release ?

Regards.
Comment 8 Peter Korsgaard 2019-10-31 22:07:36 UTC
(In reply to Jean-pierre Cartal from comment #7)

We have instead removed the check-uniq-files logic:

https://git.buildroot.org/buildroot/commit/?id=2496189a4207173e4cd5bbab90256f911175ee57