The performance of the NodeJS executable, as compiled by Buildroot, has exceptionally poor startup performance, taking around 30s to run. The prebuilt binaries from NodeJS running on the same system take less than 2s to run. I created a Gist with a comparison using the time command: https://gist.github.com/erichiggins/ac139ebe34b8d3c6dbb68f3b09ca9008 In both cases, the runtime performance is about equal. The prebuilt binaries require the buildroot system to use glibc and will not run if a more space-efficient alternative (uClibc-ng) is selected. Further, NodeJS v12.x (now the major version used by Buildroot) does not officially support ARM v6l and is considered experimental. https://github.com/nodejs/build/issues/1677#issuecomment-486204349 Current workaround is to use glibc and overwrite the NodeJS executables with the unofficial prebuilts via rootfs overlay. https://github.com/nodejs/unofficial-builds/ It would be quite helpful to build a performant NodeJS executable without requiring these workarounds. Thank you!
Are you building Buildroot with the default of BR2_OPTIMIZE_S=y (which implies using the -Os gcc optimization flag), or using BR2_OPTIMIZE_2=y or BR2_OPTIMIZE_3=y ? If you're using BR2_OPTIMIZE_S=y, could you try BR2_OPTIMIZE_2=y and BR2_OPTIMIZE_3=y and see how much it changes things ?
Yes, I'm using BR2_OPTIMIZE_S=y but I can try BR2_OPTIMIZE_2=y and report back. I emailed the listed developers for package/nodejs and Daniel emailed back with a few suggestions specific to that package's build flags, though they were not successful for me and it started to get a bit more complex than I could commit to (hence this ticket).
Hi Thomas! Long time since I've helped out with Buildroot but happy to try to be of assistance. Eric indicated to me that this largely a startup issue, and not one of node's actual performance. One of node's performance tricks is to take a snapshot of its V8 interpreter "world" and store that as a snapshot. This is accomplished using the mksnapshot command during the build. This has been tricky for them to get right when cross-compiling. Eric tried removing the --without-snapshot flag from the build but then the build failed trying to run mksnapshot (I think because it was running the target command, not the host version). Eric Said: > Maybe there's another problem in my config, but having removed that flag from nodejs.mk, the build failed. > I reverted my changes to .config, ran make clean and removed the contents from dl/ and ~/.buildroot-ccache before building. > > touch 5e8f1fa6be42bf232d2ec48bae2a475b98d6f2f6.intermediate > LD_LIBRARY_PATH=/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/lib.host:/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/obj.target/v8_snapshot/geni; "/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/mksnapshot" --turbo_instruction_scheduling "--target_os=linux" "--target_arch=arm" --startup_src "/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/obj.target/v8_snapshot/geni/embedded.S" --no-native-code-counters > /lib/ld-uClibc.so.0: No such file or directory > tools/v8_gypfiles/v8_snapshot.target.mk:16: recipe for target > '5e8f1fa6be42bf232d2ec48bae2a475b98d6f2f6.intermediate' failed > make[3]: *** [5e8f1fa6be42bf232d2ec48bae2a475b98d6f2f6.intermediate] Error 255 > ... > Makefile:101: recipe for target 'node' failed > make[2]: *** [node] Error 2 I said: > I haven't worked with this, but to me this looks like the mksnapshot command > (which has been cross-compiled for you) is now failing to run > (/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/mksnapshot). > > I found some github bug reports related to this but nothing which really is > a definitive guide. I also found this, which might help: > https://chrislea.com/2018/08/20/cross-compiling-node-js-for-arm-on-ubuntu/ . > Several of the sources I found talk about setting CC_host and CXX_host, which > to the best of my recollection the buildroot recipe does not do. IIRC > buildroot uses HOSTCC and HOSTCXX instead > (see https://buildroot.org/downloads/manual/manual.html, 8.6). So maybe that > is important? I don't know. Somewhere I also found a thing which said > that you have to be careful exactly how you set CC_host and CXX_host, to make > sure -m32 was passed. I dunno. > Buried in https://github.com/nodejs/build/issues/266 is the statement > > "I believe I have this working now and will be able to re-enable snapshots > when nodejs/node#4117 lands. V8 can handle cross compiling of snapshots if > the {CC,CXX}_host variables are defined, by compiling the mksnapshot > executable with the host compiler." > > Which seems useful. So that is my recommendation: - Double check that it's strictly a startup issue - Try to enable the snapshot feature
Using BR2_OPTIMIZE_2=y didn't make any discernible difference. time node --help ... real 0m 27.46s user 0m 22.92s sys 0m 1.90s
Is this only at first startup or does running nodejs multiple times each time take ~30s on the RPI? What storage are you using? The prebuilt nodejs presumably bundles a bunch of dependencies statically, whereas the Buildroot one uses dynamic linking
All subsequent runs of nodejs are just as slow to start. Storage is a class 10 microSD. Actually, the prebuilt nodejs binaries are also dynamically linked. I found this out the hard way because I was using uClibc-ng and the prebuilt expects glibc. (prebuilt) # file /usr/local/lib/nodejs/node-v10.18.0-linux-armv6l/bin/node /usr/local/lib/nodejs/node-v10.18.0-linux-armv6l/bin/node: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.16.42, with debug_info, not stripped (compiled by buildroot) # file /usr/bin/node /usr/bin/node: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-uClibc.so.0, stripped For what it's worth, the flags they use for armv6l are available here: https://github.com/nodejs/unofficial-builds/tree/master/recipes/armv6l
(In reply to Eric Higgins from comment #6) Yes, both builds dynamically link to libc, but the unofficial builds probably statically link to openssl / libuv / zlib / .. Thanks for the link to their Dockerfile, but I don't seem to find where they run ./configure?
I don't see them running ./configure either. My guess is that they are passing in flags and options as env variables instead?
Looks like ./configure is called a few times within the Makefile https://github.com/nodejs/node/blob/v12.14.0/Makefile
Thank you for your report. The issue tracker for the Buildroot project has been moved to the Gitlab.com issue tracker: https://gitlab.com/buildroot.org/buildroot/-/issues We are taking this opportunity to close old issues in this old tracker. If you believe your issue is still relevant, please open one in the new issue tracker. Thank you!