Bug 13141

Summary: Target-finalize fail with "depmod: ERROR: Bad version passed"
Product: buildroot Reporter: laurent
Component: OtherAssignee: unassigned
Status: RESOLVED FIXED    
Severity: normal CC: buildroot, yann.morin.1998
Priority: P5    
Version: 2020.02.1   
Target Milestone: ---   
Hardware: Other   
OS: Linux   
Host: Target:
Build:
Attachments: 0001-linux-workaround-make-4.1-bug-which-one.patch

Description laurent 2020-08-06 22:03:27 UTC
Hello,

I've successfully build my system up to 2020.02.

But when I started to try to build 2020.02.1 and onwards it always failed with "depmod: ERROR: Bad version passed" message during TARGET_FINALIZE step.

I managed to find the commit that make my build fail:

8f5a610fe81676092a6960415717de731f43f8bf
linux: Allow modprobe for out-of-tree modules

When LINUX_TARGET_FINALIZE_HOOKS are call the new LINUX_RUN_DEPMOD function fail in my case.

# Run depmod in a target-finalize hook, to encompass modules installed by
# packages.
define LINUX_RUN_DEPMOD
	$(HOST_DIR)/sbin/depmod -a -b $(TARGET_DIR) $(LINUX_VERSION_PROBED)
endef
LINUX_TARGET_FINALIZE_HOOKS += LINUX_RUN_DEPMOD

Here is my $(LINUX_VERSION_PROBED): 4.19.78-linux4sam-6.2 

I'm using a buildroot external tree for my board, still I've attached my .config file if it can help

What I don't understand is that if I run manually the depmod command it pass successfully :/

---> builroot log:
/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target `/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`

depmod: ERROR: Bad version passed make[1]:

make: *** [Makefile:747: target-finalize] Error 1


---> My manual commands (that succeed):
#/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null

4.19.78-linux4sam-6.2

#/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target 4.19.78-linux4sam-6.2

OK

Any help appreciated!
Comment 1 Yann E. MORIN 2020-08-23 22:09:26 UTC
Laurent, All,

Thanks for the report.

Could you try a few things for me, please:
 1- run:  make V=1
 2- in the LINUX_RUN_DEPMOD macro, add this as the first command:
        printf '"%s"\n' "$(LINUX_VERSION_PROBED)"
 3- try a build without ccache

Also, I noticed something weird in your report:


--> My manual commands (that succeed):
#/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null

4.19.78-linux4sam-6.2


There is an empty line between the command and the version string. Is it
really what happens, or is it an artefact of your copy-paste?

Regards,
Yann E. MORIN.
Comment 2 Asaf Kahlon 2020-08-24 05:00:48 UTC
(In reply to laurent from comment #0)

I encountered the same error few weeks ago.
I don't know if we're having the same case, but here are the results of my little research:

1. I didn't run the 'make' command from Buildroot, I had something like: "make -j8 -C buildroot/". If I run the `make` command directly from Buildroot, it works fine.
* It always happens with compiling in parallel, doesn't matter whether it's -j2 or -j100. When I compiled with just make, it worked.
* It fails only on the first time! After the error occurred, I run "make -j8 -C buildroot/" *again* and it worked.

Are those points relevant to your case?

Regards,
Asaf.
Comment 3 laurent 2020-08-24 23:33:38 UTC
Hi yann,

> make V=1:
if grep -q "CONFIG_MODULES=y" /home/buildroot/output/build/linux-linux4sam_6.2/.config; then /home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target `/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`; fi
depmod: ERROR: Bad version passed make[1]:
make: *** [Makefile:747: target-finalize] Error 1
make : on quitte le répertoire « /home/buildroot »
make : on entre dans le répertoire « /home/buildroot »

> Adding printf 'probed linux version:"%s"\n'
>
># Run depmod in a target-finalize hook, to encompass modules installed by
># packages.
>define LINUX_RUN_DEPMOD
>        printf 'probed linux version:"%s"\n' "$(LINUX_VERSION_PROBED)"
>        if grep -q "CONFIG_MODULES=y" $(LINUX_DIR)/.config; then \
>                $(HOST_DIR)/sbin/depmod -a -b $(TARGET_DIR) >$(LINUX_VERSION_PROBED); \
>        fi
>endef
>LINUX_TARGET_FINALIZE_HOOKS += LINUX_RUN_DEPMOD

printf 'probed linux version:"%s"\n' "`/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`"
probed linux version:"make[1]: Entering directory '/home/buildroot'
4.19.78-linux4sam-6.2
make[1]: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'"
if grep -q "CONFIG_MODULES=y" /home/buildroot/output/build/linux-linux4sam_6.2/.config; then /home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target `/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`; fi
depmod: ERROR: Bad version passed make[1]:
make: *** [Makefile:747: target-finalize] Error 1
make : on quitte le répertoire « /home/buildroot »
make : on entre dans le répertoire « /home/buildroot »

>Removing ccache usage
printf 'probed linux version:"%s"\n' "`/usr/bin/make  HOSTCC="/usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`"
probed linux version:"make[1]: Entering directory '/home/buildroot'
4.19.78-linux4sam-6.2
make[1]: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'"
if grep -q "CONFIG_MODULES=y" /home/buildroot/output/build/linux-linux4sam_6.2/.config; then /home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target `/usr/bin/make  HOSTCC="/usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`; fi
depmod: ERROR: Bad version passed make[1]:
make: *** [Makefile:747: target-finalize] Error 1


For the empty line, I added it to empathize the result, it was not there, you are right.

Laurent
Comment 4 laurent 2020-08-25 05:43:44 UTC
(In reply to Asaf Kahlon from comment #2)
Asaf, Yann,

1/As suggested by asaf I checked my build script and I had:
make -C $BR_SOURCE O=$BR_BUILD -j`nproc` V=1

Removing -j`nproc` seems to remove the issue.
 
--> I did the test during the night, I'll recheck tonight to be sure.

2/Asaf you are right first time I build it failed (with -j`nproc`), then next time it succeed, but I was suspecting that buildroot somehow .stamp'ed the TARGET_FINALIZE stage.

Laurent
Comment 5 Yann E. MORIN 2020-08-25 07:35:29 UTC
Laurent, All,

> printf 'probed linux version:"%s"\n' "`/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`"
> probed linux version:"make[1]: Entering directory '/home/buildroot'
> 4.19.78-linux4sam-6.2
> make[1]: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'"

OK, so the bug is already visible here : see how the output contains make messages around
the version value.

However, this is very strange, because:

1. we're telling make to not print those emssages: --no-print-directory -s
2. it is entering and leaving different directoriee: Entering directory '/home/buildroot'
   but: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'

So I suspect a weird corner-case bug in make... 

Oh, by the way, top-level parallel build is not 100% supported, though, and at the
very least, requires that you enable BR2_PER_PACKAGE_DIRECTORIES.

Regards,
Yann E. MORIN.
Comment 6 Yann E. MORIN 2020-08-25 14:30:28 UTC
Laurent, Asaf, All,

So I was initially not able to reproduce the issue, because my umask was
0002, which is not the umask Buildroot expects to work with. In this
situation, Buildroot respawns itself with a proper umask:

    https://git.buildroot.org/buildroot/tree/Makefile#n76

When I set a proper umask 0022, then I can indeed reproduce the issue,
but only in Ubuntu 18.04!

Still more investigations required...
Comment 7 Yann E. MORIN 2020-08-25 15:43:48 UTC
Created attachment 8576 [details]
0001-linux-workaround-make-4.1-bug-which-one.patch

Laurent, Asaf, All,

Could you please try to check if the attached patch solves the issue for you?

Regards,
Yann E. MORIN.
Comment 8 laurent 2020-08-25 18:57:24 UTC
Yann,
The patch is working!
I can activate all my procs :D, welcome back lightning fast compilation!

Thx!
Comment 9 Yann E. MORIN 2020-08-25 19:29:13 UTC
(In reply to laurent from comment #8)
> The patch is working!

Great, thanks for the feedback.

I am not sure this will be the final form of the fix, or rather the workaround,
so maybe we'll need more testing...

Regards,
Yann E. MORIN.
Comment 10 Peter Korsgaard 2020-08-28 21:06:41 UTC
Worked around in git:
https://git.buildroot.org/buildroot/commit/?id=3f6a40e9fae92b3fe476d82449ddd6851cea03da