Bug 10511

Could you indicate how to reproduce the problem you're seeing with the official Buildroot, and provide a minimal Buildroot .config / scenario that exhibits the issue ? Thanks!

(In reply to Thomas Petazzoni from comment #1)

This is due to a change in behavior with wget 1.19.2. This discussion describes the issue: http://lists.gnu.org/archive/html/bug-wget/2017-11/msg00000.html. To reproduce, I installed wget 1.19.2 on my system and added the following line to qemu_arm_versatile_defconfig:

BR2_PACKAGE_ERLANG=y

Here's the result of running "make source". You can see the MD5 mismatch from the erlang.org copy. That's because it was automatically decompressed by wget.

umask 0022 && make -C /home/fhunleth/buildroot.upstream O=/home/fhunleth/buildroot.upstream/o/wgettest/. source
>>> erlang 20.0 Downloading
--2017-12-18 08:46:07--  http://www.erlang.org/download/otp_src_20.0.tar.gz
Resolving www.erlang.org... 31.172.186.60
Connecting to www.erlang.org|31.172.186.60|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://erlang.org/download/otp_src_20.0.tar.gz [following]
--2017-12-18 08:46:08--  http://erlang.org/download/otp_src_20.0.tar.gz
Resolving erlang.org... 192.121.151.106
Connecting to erlang.org|192.121.151.106|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87346046 (83M) [application/x-tar]
Saving to: ‘/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.5GQ6qy/output’

/home/fhunleth/buildroot.upstream/o/ 100%[======================================================================>]  83.30M   582KB/s    in 2m 27s  

2017-12-18 08:48:35 (582 KB/s) - ‘/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.5GQ6qy/output’ saved [237383680]

ERROR: otp_src_20.0.tar.gz has wrong md5 hash:
ERROR: expected: 2faed2c3519353e6bc2501ed4d8e6ae7
ERROR: got     : 6d1210c2a3a9b178a751809016ef51d2
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
--2017-12-18 08:48:35--  http://sources.buildroot.net/otp_src_20.0.tar.gz
Resolving sources.buildroot.net... 176.9.16.109
Connecting to sources.buildroot.net|176.9.16.109|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87346046 (83M) [application/x-gzip]
Saving to: ‘/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.hY7Oae/output’

/home/fhunleth/buildroot.upstream/o/ 100%[======================================================================>]  83.30M  3.56MB/s    in 23s     

2017-12-18 08:48:59 (3.59 MB/s) - ‘/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.hY7Oae/output’ saved [87346046/87346046]

otp_src_20.0.tar.gz: OK (md5: 2faed2c3519353e6bc2501ed4d8e6ae7)
otp_src_20.0.tar.gz: OK (sha256: fe80e1e14a2772901be717694bb30ac4e9a07eee0cc7a28988724cbd21476811)
otp_src_20.0.tar.gz: OK (md5: 2faed2c3519353e6bc2501ed4d8e6ae7)
otp_src_20.0.tar.gz: OK (sha256: fe80e1e14a2772901be717694bb30ac4e9a07eee0cc7a28988724cbd21476811)

Verify that the invalidate MD5 was done on the uncompressed file:

$ gunzip otp_src_20.0.tar.gz 
$ md5sum otp_src_20.0.tar 
6d1210c2a3a9b178a751809016ef51d2  otp_src_20.0.tar

It seems like a configuration change on erlang.org could fix the problem as well, but I'm not sure that's the right answer for Buildroot.

Thanks for the investigation. This problem has been fixed/worked-around upstream by commit http://git.savannah.gnu.org/cgit/wget.git/commit/?id=047746eb765d212cbf05afac16f4bd73257d8186 which ensures that .gz and .tgz files don't get uncompressed.

However wget 1.19.2 remains affected.

Options:

 - Ignore the problem, and leave users of 1.19.2 affected by the problem.

 - Bail out if wget 1.19.2 is used, and leave it to the user to use a suitable wget version

 - Detect if wget >= 1.19.2, and pass --compression=none

Thoughts?

Thomas, Jean, All,

> - Detect if wget >= 1.19.2, and pass --compression=none

That looks like the best solution IMHO. Unfortunately, wget versions < 1.19.2
do not accept the --compression option, so we really must make it conditional.

My 2ct.

Regards,
Yann E. MORIN.

This problem only affects wget 1.19.2. In 1.19.3, a patch was merged that disables auto-decompression for files ending in .gz and .tgz, and then in 1.19.4, the auto-decompression was disabled entirely, as it was causing too many problems.

Since the problem has been fixed in dot-releases of wget, most reasonable distributions that were using 1.19.2 have probably upgraded to 1.19.3 or later.

In addition, for those stuck with wget 1.19.2, the fallback to sources.buildroot.net works (even if admittedly the bogus hashes can be scary). But considering all of this, we don't want to introduce extra complexity in Buildroot just to workaround a problem affecting only wget 1.19.2 when downloading files only from HTTP servers having a specific configuration.

Thanks for your time !