| Summary: | Finding non-relative paths in the ccache | ||
|---|---|---|---|
| Product: | buildroot | Reporter: | Mark Hirota <mhirota> |
| Component: | Other | Assignee: | Arnout Vandecappelle <arnout> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | buildroot |
| Priority: | P5 | ||
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Host: | Target: | ||
| Build: | |||
Examining the build of objects that have non-relative paths into the buildroot output directory in ccache shows how they are getting created. Here's an example from host-uboot-tools:
/work/build-system/output/r700/host/bin/ccache /usr/lib64/ccache/gcc -Wp,-MD,tools/.mkimage.o.d -O2 -I/work/build-system/output/r700/host/include -include ./include/libfdt_env.h -idirafterinclude -idirafter./arch//include -I./lib/libfdt -I./tools -DUSE_HOSTCC -D__KERNEL_STRICT_NAMES -D_GNU_SOURCE -c -o tools/mkimage.o tools/mkimage.c
Since this is building a host binary it doesn't not call the buildroot toolchain wrapper. The toolchain wrapper is what sets basedir for ccache and so ccache will not make absolute paths into the buildroot output directory tree relative.
I think in order to see this one needs to build a host binary, have the package's build system for the host binary use buildroot's cccache, have it produce an auto-dependency file (-MD, etc), and have the code use a header that in the buildroot output tree, i.e. some part of a host library package.
Perhaps this could be fixed if CCACHE_BASEDIR was injected in a way that also managed to apply it to host compilations?
This ends up being a big problem if multiple buildroot output directories are used. For example:
make O=first some_defconfig ; make O=first all
rm -rf first
make O=second some_defconfig ; make O=second all
The build in second fails because ccache will use cached dependency files from the first build, and these cached files reference headers with a "*/first/*" path and those headers no longer exist.
Arnout, since you're looking after all the ccache related topics, could you have a look at this? Thanks! Does the build really fail in that second example? Since the first/ directory is in the command-line arguments during the build and it doesn't match BR2_CCACHE_DIR, ccache should just treat it as a cache miss. So indeed, ccache doesn't work for host packages unless the output directory is the same. We could actually easily export CCACHE_BASEDIR from make, so that it works also for host packages. We could perhaps even remove it from the toolchain wrapper then, though it's probably convenient that the same basedir is used when building outside of Buildroot. That said, this problem will still exist when a BR2_HOST_DIR is specified that points outside of the output directory. In that case we'd need two different 'base directories', which is obviously not possible... Arnout: yes, we have seen a failure as Trent describes, but with 2 similar configs: r700 and r700_rescue.
>>> uboot-tools 2017.09 Building
PATH="/mnt/work/output/r700/host/bin:/mnt/work/output/r700/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" /usr/bin/make -j5 -C /mnt/work/output/r700/build/uboot-tools-2017.09 CROSS_COMPILE="/mnt/work/output/r700/host/bin/arm-buildroot-linux-gnueabihf-" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O3 -fstack-protector-strong" LDFLAGS="" STRIP=/mnt/work/output/r700/host/bin/arm-buildroot-linux-gnueabihf-strip CROSS_BUILD_TOOLS=y tools-only
CHK include/config/uboot.release
CHK include/generated/timestamp_autogenerated.h
HOSTCC scripts/basic/fixdep
UPD include/generated/timestamp_autogenerated.h
UPD include/config/uboot.release
CHK include/generated/version_autogenerated.h
UPD include/generated/version_autogenerated.h
HOSTCC tools/mkenvimage.o
WRAP tools/lib/crc32.c
HOSTCC tools/os_support.o
HOSTCC tools/aisimage.o
HOSTCC tools/atmelimage.o
WRAP tools/common/bootm.c
HOSTCC tools/default_image.o
fixdep: error opening config file: /mnt/work/output/r700_rescue/host/lib/gcc/arm-buildroot-linux-gnueabihf/7.2.0/include/stddef.h: No such file or directory
Our build automation performs clean builds, but shares a buildroot-ccache across them. This type of failure forces us to build the projects on separate build hosts to avoid hitting it.
Hm, this build failure doesn't look like it will be solved with CCACHE_BASEDIR. Indeed, it is the *target* ccache'd compiler that generated a *.d file that apparently still has a full path in it. I don't see how this could happen... Possibly the issue is that this is a path that is embedded in the specs file, and ccache will only substitute paths that get passed on the command line (through -I etc.). Needs more debugging... Arnout: I believe the issue is with ccache 3.3.4. I see in the 3.3.5 release notes (https://ccache.samba.org/releasenotes.html#_ccache_3_3_5) the following: > Fixed a bug related to erroneously storing a dependency file with absolute paths in the cache on a preprocessed hit. Should the ccache version in buildroot be updated to 3.3.5? Good catch! Can you try updating ccache, check if that fixes things for you, and if so send the patch to the list with a reference to this bug in the commit log? Fixed in git by bumping ccache to 3.3.5 I seems like Arnout should be right about host compiles in different directories not sharing ccache. Since CCACHE_BASEDIR is not set, the paths should be absolute and different directories would be a cache miss. But here's some testing with host-uboot-tools. First a build from a cleared and reset ccache: cache hit (direct) 0 cache hit (preprocessed) 0 cache miss 58 cache hit rate 0.00 % called for link 8 cleanups performed 0 files in cache 171 As expected, all misses. 58 files. Building again from the same output dir: cache hit (direct) 53 cache hit (preprocessed) 5 cache miss 58 cache hit rate 50.00 % called for link 16 cleanups performed 0 files in cache 176 Exactly 58 new hits, as expected, everything in cache. But 5 new files, odd. Now the tricky one, building from a different output dir: cache hit (direct) 53 cache hit (preprocessed) 15 cache miss 106 cache hit rate 39.08 % called for link 24 cleanups performed 0 files in cache 331 These should be all misses, right? But there were 10 hits in preprocessed mode! Which is fair, if the preprocessed code ends up being exactly the same. But just because the c code is the same doesn't mean the auto-dependency file is the same (absolute paths in the .d file differ!). If ccache tries to use the cached .d file with absolute paths on this supposed preprocessed hit, it will fail. |
It appears that non-relative paths are leaking into the buildroot ccache despite the default BR2_CCACHE_USE_BASEDIR=y. Here are my steps to demonstrate this: $ git clone git://git.buildroot.net/buildroot $ cd buildroot $ make zynq_zybo_defconfig $ make menuconfig $ # Build options --> $ # [*] Enable compiler cache $ # < Save > and < Exit > back to prompt $ make At this point, I grep the ccache for anything that starts with "/home": $ grep -r --include "*.d" "^\/home" $HOME/.buildroot-ccache/ | wc -l 647 I was expecting this to be zero. Just to confirm some assumptions: $ grep -r --include "*.d" "^\.\." $HOME/.buildroot-ccache/ | wc -l 60840 $ grep CCACHE .config BR2_CCACHE=y BR2_CCACHE_DIR="$(HOME)/.buildroot-ccache" BR2_CCACHE_INITIAL_SETUP="" BR2_CCACHE_USE_BASEDIR=y # BR2_PACKAGE_CCACHE is not set