Bug 6086

Summary:	tar is broken on Android: resulting stream is too short and loses information
Product:	Busybox	Reporter:	yuri
Component:	Standard Compliance	Assignee:	unassigned
Status:	RESOLVED FIXED
Severity:	critical	CC:	busybox-cvs
Priority:	P5
Version:	unspecified
Target Milestone:	---
Hardware:	PC
OS:	Other
Host:		Target:
Build:

Description yuri 2013-03-10 21:52:30 UTC

I have a directory on Android OS, on sdcard, with the 2.4GB file in it.
'ls' indicates that file is 2.4GB, yet resulting tar stream is only 1.8GB, as measured by 'wc', see below. Also, unpacking this tar (on host machine) produces the file with the same size (2.4GB) but with damaged content (sha256 checksum mismatch). (Of course, tar archive should always be larger than the total of sizes of all files in it.)

# /system/xbin/tar cf - Xxxxx\ Xxxxx\ \(XXXX\)\ \[xxxxx\] | wc
  7163411  40184336 1823790164

# ls -l  Xxxxx\ Xxxxx\ \(XXXX\)\ \[xxxxx\]/
total 4827041
----rwxr-x    1 system   sdcard_r 2471310806 Mar  8 16:44 Xxxxx.Xxxxx.XXXX.XXXXX.Xxxxx.xxxx.XXXX.xxx
----rwxr-x    1 system   sdcard_r    130677 Mar  7 15:15 XXXXXXXX.XXX.jpg


I totally can't understand how this is even possible, hence this bug.
This wouldn't be possible on any stable Linux/BSD system with any stable version of system utilities.
But there is some foul play at work here, that causes this issue.

So now, as a result, I can't transfer this directory verbatim using tar. I have to use workaround with individual files like this: 'cat one-file | ssh user@host "cat > one-file"'.

Both of these versions exhibit this problem:
BusyBox v1.20.2-jb bionic (2013-02-22 15:57 +0100) multi-call binary.
BusyBox v1.18.5 (2011-07-20 21:05:55 CDT) multi-call binary.

Comment 1 Denys Vlasenko 2013-03-14 23:57:52 UTC

(In reply to comment #0)
> I have a directory on Android OS, on sdcard, with the 2.4GB file in it.
> 'ls' indicates that file is 2.4GB, yet resulting tar stream is only 1.8GB, as
> measured by 'wc', see below.

This can be true (meaning: I can imagine a bug which does this). But...

> Also, unpacking this tar (on host machine)
> produces the file with the same size (2.4GB)

...*this* would be quite strange: raw tar with no compression can't possibly create a file which is bigger than the tarball.

Which tar do you use on host machine? GNU? If not, try GNU tar too. Does it do the same?

What "file TARBALL.tar" says? what "hexdump -C TARBALL.tar | head" show?

Comment 2 Danila Galimov 2013-07-30 19:57:50 UTC

I have the same(or maybe not the same) problem with decompressing tar.gz files on Android. I'm using newest BusyBox 1.21.1.

When I'm trying to unpack the file, it says

1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz
tar -xvZf joe-3.7-android.tar.gz
tar: invalid magic
tar: short read

Please find the hexdump attached

shell@android:/sdcard/flash $ hexdump -C joe-3.7-android.tar.gz | head
hexdump -C joe-3.7-android.tar.gz | head
00000000  1f 8b 08 00 04 ed 50 4b  00 03 ec 9a 0d 98 55 55  |......PK......UU|
00000010  b9 c7 17 30 0c 87 71 54  b0 d1 48 30 27 43 2f e9  |...0..qT..H0'C/.|
00000020  24 33 c3 08 a8 94 98 a8  a8 63 8d 8a 36 2a 79 38  |$3.......c..6*y8|
00000030  73 ce 9e 39 87 39 5f 9c  bd 0f 03 3a 25 16 1a f9  |s..9.9_....:%...|
00000040  51 54 54 68 58 64 98 68  98 9a 68 94 94 dc 61 3e  |QTThXd.h..h...a>|
00000050  e8 46 c6 4d 2b 6e 5a 72  6f 53 97 ba 54 52 d4 a5  |.F.M+nZroS..TR..|
00000060  1b 69 bf 77 af 35 b3 5f  e6 31 6f 3d 4f fa dc e7  |.i.w.5._.1o=O...|
00000070  b9 9d 87 3f 6b fd f7 bb  3e de f5 be ef fa da 7b  |...?k...>......{|
00000080  a6 a7 12 41 62 ba 79 55  7f f5 fc 66 cd 3a c3 a6  |...Ab.yU...f.:..|
00000090  33 6d da d0 58 1f a6 ee  67 1a 1a 66 34 36 35 cc  |3m..X...g..f465.|

Comment 3 Danila Galimov 2013-07-30 20:00:37 UTC

The file I'm trying to unpack can be downloaded from
http://www.konstantin-schauwecker.de/files/joe-3.7-android.tar.gz (http://www.konstantin-schauwecker.de/joe-android.php page)

Comment 4 Danila Galimov 2013-08-11 14:21:08 UTC

Btw, if works OK if i first try to unzip it using
gzip -d gzip -d joe-3.7-android.tar.gz (both /system/bin/gzip and /system/xbin/gzip works)
and when executing 
tar -xvf joe-3.7-android.tar 

So there is a problem somewhere in the gzip -> tar conjunction.

Comment 5 Denys Vlasenko 2013-08-12 00:40:52 UTC

(In reply to comment #2)
> I have the same(or maybe not the same) problem with decompressing tar.gz files
> on Android. I'm using newest BusyBox 1.21.1.
> 
> When I'm trying to unpack the file, it says
> 
> 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz
> tar -xvZf joe-3.7-android.tar.gz
> tar: invalid magic
> tar: short read

-Z is for auto-decompress, no auto-gunzip
> 
> Please find the hexdump attached
> 
> shell@android:/sdcard/flash $ hexdump -C joe-3.7-android.tar.gz | head
> hexdump -C joe-3.7-android.tar.gz | head
> 00000000  1f 8b 08 00 04 ed 50 4b  00 03 ec 9a 0d 98 55 55  |......PK......UU|
            ^^^^^

This file is gzipped, not compressed:

        COMPRESS_MAGIC = 0x9d1f,
        GZIP_MAGIC  = 0x8b1f,

Comment 6 Danila Galimov 2013-08-17 13:41:00 UTC

(In reply to comment #5)
> (In reply to comment #2)
> > I have the same(or maybe not the same) problem with decompressing tar.gz files
> > on Android. I'm using newest BusyBox 1.21.1.
> > 
> > When I'm trying to unpack the file, it says
> > 
> > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz
> > tar -xvZf joe-3.7-android.tar.gz
> > tar: invalid magic
> > tar: short read
> 
> -Z is for auto-decompress, no auto-gunzip
So, as i understand, auto g-unzip is just not implemented yet? Is it planned to be implemented?

Comment 7 Denys Vlasenko 2013-08-18 14:35:33 UTC

(In reply to comment #6)
> > > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz
> > > tar -xvZf joe-3.7-android.tar.gz
> > > tar: invalid magic
> > > tar: short read
> > 
> > -Z is for auto-decompress, no auto-gunzip
> So, as i understand, auto g-unzip is just not implemented yet? Is it planned to
> be implemented?

It is implemented:

$ busybox tar --help
BusyBox v1.22.0.git (2013-07-30 11:58:43 CEST) multi-call binary.

Usage: tar -[cxtZzJjahmvO] [-X FILE] [-T FILE] [-f TARFILE] [-C DIR] [FILE]...

Create, extract, or list files from a tar file

Operation:
	c	Create
	x	Extract
	t	List
	f	Name of TARFILE ('-' for stdin/out)
	C	Change to DIR before operation
	v	Verbose
	Z	(De)compress using compress
	z	(De)compress using gzip
	J	(De)compress using xz
	j	(De)compress using bzip2
	a	(De)compress using lzma
	O	Extract to stdout
	h	Follow symlinks
	m	Don't restore mtime
	exclude	File to exclude
	X	File with names to exclude
	T	File with names to include

Comment 8 gotrunks 2013-08-18 16:41:40 UTC

Danila, try with tar -xvzf joe-3.7-android.tar.gz instead.

tar -xvf joe-3.7-android.tar.gz may work as well

Comment 9 yuri 2013-08-21 16:09:56 UTC

Denys,

You closed this PR with RESOLVED/INVALID.
I don't see any note why this is an invalid problem.

Latest discussion was about gzipped streams, how they are implemented or not.

My OP was about unzipped stream. (!!)

Please explain why closed this PR.

Regards,
Yuri

Comment 10 Danila Galimov 2013-09-21 13:38:27 UTC

(In reply to comment #7)
> (In reply to comment #6)
> > > > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz
> > > > tar -xvZf joe-3.7-android.tar.gz
> > > > tar: invalid magic
> > > > tar: short read
> > > 
> > > -Z is for auto-decompress, no auto-gunzip
> > So, as i understand, auto g-unzip is just not implemented yet? Is it planned to
> > be implemented?
> 
> It is implemented:
> 
> $ busybox tar --help
> BusyBox v1.22.0.git (2013-07-30 11:58:43 CEST) multi-call binary.
> 
> Usage: tar -[cxtZzJjahmvO] [-X FILE] [-T FILE] [-f TARFILE] [-C DIR] [FILE]...
> 
> Create, extract, or list files from a tar file
> 
> Operation:
>     c    Create
>     x    Extract
>     t    List
>     f    Name of TARFILE ('-' for stdin/out)
>     C    Change to DIR before operation
>     v    Verbose
>     Z    (De)compress using compress
>     z    (De)compress using gzip
>     J    (De)compress using xz
>     j    (De)compress using bzip2
>     a    (De)compress using lzma
>     O    Extract to stdout
>     h    Follow symlinks
>     m    Don't restore mtime
>     exclude    File to exclude
>     X    File with names to exclude
>     T    File with names to include

Denis, my version shows that slightly another options:

1|shell@android:/storage/sdcard0/flash $ busybox tar --help
busybox tar --help
BusyBox v1.21.1-Stericson (2013-07-08 15:58:11 BST) multi-call binary.

Usage: tar -[cxtZhmvO] [-X FILE] [-T FILE] [-f TARFILE] [-C DIR] [FILE]...

Create, extract, or list files from a tar file

Operation:
        c       Create
        x       Extract
        t       List
        f       Name of TARFILE ('-' for stdin/out)
        C       Change to DIR before operation
        v       Verbose
        Z       (De)compress using compress
        O       Extract to stdout
        h       Follow symlinks
        m       Don't restore mtime
        exclude File to exclude
        X       File with names to exclude
        T       File with names to include

Maybe it was included only in 1.22 version?

Comment 11 Danila Galimov 2013-09-21 13:41:02 UTC

(In reply to comment #8)
> Danila, try with tar -xvzf joe-3.7-android.tar.gz instead.
> 
> tar -xvf joe-3.7-android.tar.gz may work as well

It doesn't work with both cases, saying
"1|shell@android:/storage/sdcard0/flash $ tar -xvzf joe-3.7-android.tar.gz
tar -xvzf joe-3.7-android.tar.gz
tar: invalid option -- z" for the first
and or that earlier referenced output about invalid magic

Comment 12 Danila Galimov 2013-09-26 14:26:22 UTC

(In reply to comment #11)
> (In reply to comment #8)
> > Danila, try with tar -xvzf joe-3.7-android.tar.gz instead.
> > 
> > tar -xvf joe-3.7-android.tar.gz may work as well
> 
> It doesn't work with both cases, saying
> "1|shell@android:/storage/sdcard0/flash $ tar -xvzf joe-3.7-android.tar.gz
> tar -xvzf joe-3.7-android.tar.gz
> tar: invalid option -- z" for the first
> and or that earlier referenced output about invalid magic

I've contacted the author of this busybox build, Stephen Erickson, and he have solved the problem. It seems that tar was not build with gzip option enabled.