Bug 6302

I'm no buildroot or github expert (so apologies if I get some of the terminology wrong), but I think I've found a bit of a nasty problem with the way buildroot downloads specific versions of packages from github.com. At first I thought it was because the .mk files for the specific rpi-* packages I was looking at were set up wrong, but I've been investigating the problem for much of this afternoon and I think it actually affects the .mk files for many packages :-(

According to http://www.buildroot.org/downloads/manual/manual.html#github-download-url packages in github with a specific tag can be downloaded with:
FOO_VERSION = v1.0
FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
(where here 'v1.0' is the tag in github)

And for those packages where you don't have a specific tag but instead want to download a specific revision, you can use:
FOO_VERSION = 1234567
FOO_SITE = http://github.com/<user>/<package>/tarball/<branch>
(where here '1234567' is the first 7 characters of the github commit hash)

In my tests, the former (downloading a specific tag) works okay, but the latter (downloading a specific revision) actually doesn't work as expected. I can't tell if this used to work and has become broken recently by a change to github.com, or if it's always been broken (but nobody ever noticed), or if I'm simply misunderstanding things (to save me looking stupid, hopefully it's not this last one!).

I'll stick with the two FOO examples above, but substituting 'fooauthor' for '<user>', 'foopackage' for '<package>' and 'master' for <branch>...

When downloading a specific tag, the first example above would download http://github.com/fooauthor/foopackage/tarball/v1.0/foo-v1.0.tar.gz to dl/foo-v1.0.tar.gz and this is the part that works correctly.
When downloading a specific revision, the second example above would download http://github.com/fooauthor/foopackage/tarball/master/foo-1234567.tar.gz to dl/foo-1234567.tar.gz and this appears to work fine, but it actually downloads the latest revision of the master branch, and not the 1234567 revision as you might expect! So if 1234567 _is_ indeed the latest revision, then things all look okay, which may be why nobody noticed this bug before, but things go quickly wrong if there's commits after 1234567.

In my testing, it seems that _whatever_ filename you have after the .../master/ in the URL will get ignored, and you'll always get the latest commit from the master branch. However it seems that you can actually specify a revision instead of <branch> and it will then work as expected, i.e. AFAICT the 'download by revision' example above should be modified to:
FOO_VERSION = 1234567
FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)

and then buildroot will download http://github.com/fooauthor/foopackage/tarball/1234567/foo-1234567.tar.gz to dl/foo-1234567.tar.gz and this *will* then work as expected. Obviously you could set FOO_VERSION to 'master' and it would go back to the previous behaviour of downloading the latest commit on the master branch, but at least this behaviour would then be explicit rather than implicit.

Where the <branch> has been used (which I've replaced with $(FOO_VERSION)) I've found that you can either use a tag-name, a branch-name, or a (full or partial) commit-id, and then following another slash you can include any filename you want, which makes no difference to the file github.com serves to you.

To give a more concrete example, I found a random repository with lots of commits, tags and branches: https://github.com/libgit2/libgit2
As of 2012-06-06 these URLs all result in the identical file being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/master  (wget saves to 'master')
http://github.com/libgit2/libgit2/tarball/master/  (wget saves to 'index.html')
http://github.com/libgit2/libgit2/tarball/master/libgit2-master.tar.gz  (wget saves to 'libgit2-master.tar.gz')
http://github.com/libgit2/libgit2/tarball/master/libgit2-dummy-filename.tgz
[by tag-name]
http://github.com/libgit2/libgit2/tarball/v0.18.0
http://github.com/libgit2/libgit2/tarball/v0.18.0/libgit2-v0.18.0.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/a50086d
http://github.com/libgit2/libgit2/tarball/a50086d/libgit2-a50086d.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174/libgit2-a50086d174.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b/libgit2-a50086d174658914d4d6462afbc83b02825b1f5b.tar.gz
(I used 'sha1sum *' to verify all the downloaded files are identical. In each case the files inside the tarfile are inside a 'libgit2-libgit2-a50086d' subdirectory)

And these URLs result in identical files being downloaded:
[by commit-id]
http://github.com/libgit2/libgit2/tarball/36c2dfe
http://github.com/libgit2/libgit2/tarball/36c2dfe/libgit2-36c2dfe.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed69/libgit2-36c2dfed69.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed696f80a20ca1352f32ec8b136b800c30/libgit2-36c2dfed696f80a20ca1352f32ec8b136b800c30.tar.gz

And these URLs result in identical files being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/development
http://github.com/libgit2/libgit2/tarball/development/libgit2-development.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/2e1fa15
http://github.com/libgit2/libgit2/tarball/2e1fa15/libgit2-2e1fa15.tar.gz

etc. etc. you get the idea. The only (unlikely occurrence) I didn't test was a branch-name having the same name as a tag-name. Obviously the match-ups between branch-names and commit-ids shown above will go 'out of sync' once more commits have been made to those branches.


In summary, I believe http://www.buildroot.org/downloads/manual/manual.html#github-download-url should be updated to say that FOO_SITE should always look like:
FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
and then mention that FOO_VERSION can either specify a tag-name or a commit-id or a branch-name.

In the current git version of buildroot, by constructing a funky find command:
find package -exec grep -P '^[A-Z0-9_]+_SITE\s*=\s*https?://github\.com/.*/tarball/(?!.*\$\(.+\))' {} \; -print | grep \.mk | sort

...I believe these are the packages that need to have their _SITE setting updated in their .mk file:
jsmin, libcofi, linenoise, lua-msgpack-native, mtdev2tuio, ne10, omap-u-boot-utils, rpi-firmware, rpi-userland, socketcand, sunxi-boards, sunxi-tools, ti-utils, tslib