The top level Makefile in buildroot has a recursive rule which causes the appearance of a hang as the number of directories in BR2_EXTERNAL increases. When the number of directories in BR2_EXTERNAL is small, the recursion occurs, but make detects the recursion and determines the target does not have to be remade. This allows make to progress. This is the failing rule: define percent_defconfig # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig %_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile @$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \ $$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN) endef $(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call percent_defconfig,$(d))$(sep))) The rule for %defconfig is created for each directory in BR2_EXTERNAL. When the rule is matched, the stem is 'defconfig_name'. The second prerequisite is expanded to $(1)/configs/defconfig_name_defconfig. The rule, and all of the other rules defined by this macro, are invoked again, but the stem is now $(1)/configs/defconfig_name_defconfig. The second prerequisite is now expanded to $(1)/configs/($1)/configs/defconfig_name_defconfig. This expansion continues until make detects the infinite recursion.
David, All, Thanks for the report, and the detailed explanations (which I did not yet entirely groked, and will need a bit more time before I see the issue). How many br2-external trees are you using? I've done a bit of testing with 0 to 1000 br2-external trees, and here are the results: $ time make BR2_EXTERNAL='' list-defconfigs real 0m0.129s $ time make BR2_EXTERNAL='(1 item)' list-defconfigs real 0m0.138s $ time make BR2_EXTERNAL='(10 item)' list-defconfigs real 0m0.262s $ time make BR2_EXTERNAL='(100 item)' list-defconfigs real 0m1.912s $ time make BR2_EXTERNAL='(1000 item)' list-defconfigs real 0m19.344s So, what we can see, is that it is roughly linear to the number of br2-external trees. Even the 1000-tree case is far from appearing stuck (granted, it is a bit long, but after seeing 2s for 100 items, it did not feel excruciating to wait). Also, I do not have any message from make 4.2.1 that reports any infinite recursion. Can you provide a bit more details on your configuration? Regards, Yann E. MORIN.
I am sorry, I am working on a github project to show the problem. Do you have a defconfig file in any of the external projects? It should only take one. I am concerned also about the name of the defconfig file.
David, All, Here's how I prepared the test br2-external trees: $ for i in $(seq 1 1000); do mkdir -p br2-external-${i}/configs touch br2-external-${i}/{Config.in,external.mk} echo "name: BR_TEST_${i}" >br2-external-${i}/external.desc touch br2-external-${i}/configs/foo{,_${i}}_defconfig done So, I get minimal br2-external trees, each with two defconfigs: one that is present in all br2external trees, and one that is specific to each tree. And here is exactly how I tested that: $ make --version GNU Make 4.2.1 $ for ext in 1 10 100 1000; do time make BR2_EXTERNAL="$(for i in $(seq 1 ${ext}); do echo $(pwd)/br2-external-${i}; done)" list-defconfigs >/dev/null; done 2>&1 |grep real real 0m0.179s real 0m0.245s real 0m1.299s real 0m18.277s What version of make are you using? How many br2-external trees do you have? However, please do note that indeed, it was never envisioned that more than a few br2-external trees be used simultaneously. But that it feels like it got stuck is still not a nice behaviour... Regards, Yann E. MORIN.
David, All, To be noted: on my machine, in the directory I was using, I was limited to using just 2446 br2-external trees, otherwise, make would whine with: - with 2447: make: execvp: /bin/sh: Argument list too long - with 2248 and above: -bash: /usr/bin/make: Argument list too long Yes, note how 2446 succeeds, 2447 has a special error, and 2448 and above have another error... Meh... But still, 2446 br2-external trees is clearly way more than was ever expected when the feature was added... back in 2016! Regards, Yann E. MORIN.
Here is a simple project that shows the problem: https://github.com/dclawso/buldroot-recursive
This shows the recursion occurs for only one external package, but it is largely benign as make detects the recursion and determines the target does not have to be remade. However, with as few as 6 external packages, the number of dependency checks that are performed causes make to hang for several hours. We currently are at 8 external packages and I left the compile running over the weekend with no progress.
David, All, OK, I could reproduce the issue with a much less convoluted example than your script. The issue does only lanifests itself when actually trying to configure Buildroot. Listing the defconfig like I was doing was not enough to trigger the issue. What's weird, is that using 5 external trees does not exhibit the issue; it is very fast. But using 6 trees triggers the issue, and make is insanely long (I did not even wait for it to finish, in fact). Note however that your comments are a bit confusing. You are talking about external packages, but that is incorrect; it is about _br2-external trees_. Also, your script is very confusing, because it does very convoluted things (instead of providing a git tree that was already fully prepared). But since we don't need it to reproduce the issue (and thus to test a fix), I'm not going to comment further. ;-) I'll lok into devising a fix... Regards, Yann E. MORIN.
David, All, Could you please have a go with, and test that patch, please: https://patchwork.ozlabs.org/project/buildroot/patch/20220917073445.2536513-1-yann.morin.1998@free.fr/ Regards, Yann E. MORIN.
The comments for $1 and $2 appear to be reversed. I am still attempting to incorporate this patch into the build tree I am using.
This change is working in my environment. Thank you!
David, All, (In reply to David Lawson from comment #10) > This change is working in my environment. Thanks for the feedback. I've just resent a v3 with trivial changes. If you can validate it still works, could you please reply to the mail with your review tag, ple
David, All, (In reply to David Lawson from comment #10) > This change is working in my environment. Thanks for the feedback. I've just resent a v3 with trivial changes. If you can validate it still works, could you please reply to the mail with your tested tag, please: Tested-by: Your NAME <your@email> Regards, Yann E. MORIN. PS. Sorry for the previous spurious message, PEBCAK.
David, All, Thank you for the report. We believe this is now fixed with commit e6195c53041f (Makefile: fix use of many br2-external trees). If you still encounter the issue, please re-open. Regards, Yann E. MORIN.