Bug 14996

Summary: Too many directories in BR2_EXTERNAL causes hang during make
Product: buildroot Reporter: David Lawson <david.lawson1>
Component: OtherAssignee: Yann E. MORIN <yann.morin.1998>
Status: RESOLVED FIXED    
Severity: normal CC: buildroot, yann.morin.1998
Priority: P5    
Version: unspecified   
Target Milestone: ---   
Hardware: All   
OS: Linux   
Host: ubuntu 22.04 Target:
Build:

Description David Lawson 2022-09-16 13:58:57 UTC
The top level Makefile in buildroot has a recursive rule which causes the appearance of a hang as the number of directories in BR2_EXTERNAL increases. When the number of directories in BR2_EXTERNAL is small, the recursion occurs, but make detects the recursion and determines the target does not have to be remade. This allows make to progress.

This is the failing rule:

define percent_defconfig
# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
endef
$(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call percent_defconfig,$(d))$(sep)))

The rule for %defconfig is created for each directory in BR2_EXTERNAL. When the rule is matched, the stem is 'defconfig_name'. The second prerequisite is expanded to $(1)/configs/defconfig_name_defconfig. The rule, and all of the other rules defined by this macro, are invoked again, but the stem is now $(1)/configs/defconfig_name_defconfig. The second prerequisite is now expanded to $(1)/configs/($1)/configs/defconfig_name_defconfig. This expansion continues until make detects the infinite recursion.
Comment 1 Yann E. MORIN 2022-09-16 16:16:32 UTC
David, All,

Thanks for the report, and the detailed explanations (which I did not yet entirely
groked, and will need a bit more time before I see the issue).

How many br2-external trees are you using? I've done a bit of testing
with 0 to 1000 br2-external trees, and here are the results:

$ time make BR2_EXTERNAL='' list-defconfigs
real    0m0.129s

$ time make BR2_EXTERNAL='(1 item)' list-defconfigs
real    0m0.138s

$ time make BR2_EXTERNAL='(10 item)' list-defconfigs
real    0m0.262s

$ time make BR2_EXTERNAL='(100 item)' list-defconfigs
real    0m1.912s

$ time make BR2_EXTERNAL='(1000 item)' list-defconfigs
real    0m19.344s

So, what we can see, is that it is roughly linear to the number of br2-external trees. Even the 1000-tree case is far from appearing stuck (granted, it is a bit
long, but after seeing 2s for 100 items, it did not feel excruciating to wait).

Also, I do not have any message from make 4.2.1 that reports any infinite recursion.

Can you provide a bit more details on your configuration?

Regards,
Yann E. MORIN.
Comment 2 David Lawson 2022-09-16 16:53:06 UTC
I am sorry, I am working on a github project to show the problem. Do you have a defconfig file in any of the external projects? It should only take one. I am concerned also about the name of the defconfig file.
Comment 3 Yann E. MORIN 2022-09-16 17:39:34 UTC
David, All,

Here's how I prepared the test br2-external trees:

$ for i in $(seq 1 1000); do
    mkdir -p br2-external-${i}/configs
    touch br2-external-${i}/{Config.in,external.mk}
    echo "name: BR_TEST_${i}" >br2-external-${i}/external.desc
    touch br2-external-${i}/configs/foo{,_${i}}_defconfig
done

So, I get minimal br2-external trees, each with two defconfigs:
one that is present in all br2external trees, and one that is
specific to each tree.

And here is exactly how I tested that:

$ make --version
GNU Make 4.2.1

$ for ext in 1 10 100 1000; do time make BR2_EXTERNAL="$(for i in $(seq 1 ${ext}); do echo $(pwd)/br2-external-${i}; done)" list-defconfigs >/dev/null; done 2>&1 |grep real
real    0m0.179s
real    0m0.245s
real    0m1.299s
real    0m18.277s

What version of make are you using? How many br2-external trees do you have?

However, please do note that indeed, it was never envisioned that more than a few
br2-external trees be used simultaneously. But that it feels like it got stuck is
still not a nice behaviour...

Regards,
Yann E. MORIN.
Comment 4 Yann E. MORIN 2022-09-16 17:56:11 UTC
David, All,

To be noted: on my machine, in the directory I was using, I was limited
to using just 2446 br2-external trees, otherwise, make would whine with:

- with 2447:
  make: execvp: /bin/sh: Argument list too long

- with 2248 and above:
  -bash: /usr/bin/make: Argument list too long

Yes, note how 2446 succeeds, 2447 has a special error, and 2448 and above
have another error... Meh...

But still, 2446 br2-external trees is clearly way more than was ever
expected when the feature was added... back in 2016!

Regards,
Yann E. MORIN.
Comment 5 David Lawson 2022-09-16 18:40:34 UTC
Here is a simple project that shows the problem:

https://github.com/dclawso/buldroot-recursive
Comment 6 David Lawson 2022-09-16 18:45:27 UTC
This shows the recursion occurs for only one external package, but it is largely benign as make detects the recursion and determines the target does not have to be remade.

However, with as few as 6 external packages, the number of dependency checks that are performed causes make to hang for several hours.

We currently are at 8 external packages and I left the compile running over the weekend with no progress.
Comment 7 Yann E. MORIN 2022-09-17 06:54:35 UTC
David, All,

OK, I could reproduce the issue with a much less convoluted example than your script.

The issue does only lanifests itself when actually trying to configure Buildroot.
Listing the defconfig like I was doing was not enough to trigger the issue.

What's weird, is that using 5 external trees does not exhibit the issue; it is
very fast. But using 6 trees triggers the issue, and make is insanely long (I did
not even wait for it to finish, in fact).

Note however that your comments are a bit confusing. You are talking about external
packages, but that is incorrect; it is about _br2-external trees_. Also, your script
is very confusing, because it does very convoluted things (instead of providing a git
tree that was already fully prepared). But since we don't need it to reproduce the
issue (and thus to test a fix), I'm not going to comment further. ;-)

I'll lok into devising a fix...

Regards,
Yann E. MORIN.
Comment 8 Yann E. MORIN 2022-09-17 07:36:03 UTC
David, All,

Could you please have a go with, and test that patch, please:
https://patchwork.ozlabs.org/project/buildroot/patch/20220917073445.2536513-1-yann.morin.1998@free.fr/

Regards,
Yann E. MORIN.
Comment 9 David Lawson 2022-09-19 17:27:30 UTC
The comments for $1 and $2 appear to be reversed. I am still attempting to incorporate this patch into the build tree I am using.
Comment 10 David Lawson 2022-09-19 21:06:05 UTC
This change is working in my environment.

Thank you!
Comment 11 Yann E. MORIN 2022-09-20 19:58:02 UTC
David, All,

(In reply to David Lawson from comment #10)
> This change is working in my environment.

Thanks for the feedback. I've just resent a v3 with trivial changes.
If you can validate it still works, could you please reply to the mail
with your review tag, ple
Comment 12 Yann E. MORIN 2022-09-20 20:00:32 UTC
David, All,

(In reply to David Lawson from comment #10)
> This change is working in my environment.

Thanks for the feedback. I've just resent a v3 with trivial changes.
If you can validate it still works, could you please reply to the mail
with your tested tag, please:

    Tested-by: Your NAME <your@email>

Regards,
Yann E. MORIN.

PS. Sorry for the previous spurious message, PEBCAK.
Comment 13 Yann E. MORIN 2023-01-07 20:11:48 UTC
David, All,

Thank you for the report.

We believe this is now fixed with commit e6195c53041f (Makefile: fix use
of many br2-external trees).

If you still encounter the issue, please re-open.

Regards,
Yann E. MORIN.