I have a directory on Android OS, on sdcard, with the 2.4GB file in it. 'ls' indicates that file is 2.4GB, yet resulting tar stream is only 1.8GB, as measured by 'wc', see below. Also, unpacking this tar (on host machine) produces the file with the same size (2.4GB) but with damaged content (sha256 checksum mismatch). (Of course, tar archive should always be larger than the total of sizes of all files in it.) # /system/xbin/tar cf - Xxxxx\ Xxxxx\ \(XXXX\)\ \[xxxxx\] | wc 7163411 40184336 1823790164 # ls -l Xxxxx\ Xxxxx\ \(XXXX\)\ \[xxxxx\]/ total 4827041 ----rwxr-x 1 system sdcard_r 2471310806 Mar 8 16:44 Xxxxx.Xxxxx.XXXX.XXXXX.Xxxxx.xxxx.XXXX.xxx ----rwxr-x 1 system sdcard_r 130677 Mar 7 15:15 XXXXXXXX.XXX.jpg I totally can't understand how this is even possible, hence this bug. This wouldn't be possible on any stable Linux/BSD system with any stable version of system utilities. But there is some foul play at work here, that causes this issue. So now, as a result, I can't transfer this directory verbatim using tar. I have to use workaround with individual files like this: 'cat one-file | ssh user@host "cat > one-file"'. Both of these versions exhibit this problem: BusyBox v1.20.2-jb bionic (2013-02-22 15:57 +0100) multi-call binary. BusyBox v1.18.5 (2011-07-20 21:05:55 CDT) multi-call binary.
(In reply to comment #0) > I have a directory on Android OS, on sdcard, with the 2.4GB file in it. > 'ls' indicates that file is 2.4GB, yet resulting tar stream is only 1.8GB, as > measured by 'wc', see below. This can be true (meaning: I can imagine a bug which does this). But... > Also, unpacking this tar (on host machine) > produces the file with the same size (2.4GB) ...*this* would be quite strange: raw tar with no compression can't possibly create a file which is bigger than the tarball. Which tar do you use on host machine? GNU? If not, try GNU tar too. Does it do the same? What "file TARBALL.tar" says? what "hexdump -C TARBALL.tar | head" show?
I have the same(or maybe not the same) problem with decompressing tar.gz files on Android. I'm using newest BusyBox 1.21.1. When I'm trying to unpack the file, it says 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz tar -xvZf joe-3.7-android.tar.gz tar: invalid magic tar: short read Please find the hexdump attached shell@android:/sdcard/flash $ hexdump -C joe-3.7-android.tar.gz | head hexdump -C joe-3.7-android.tar.gz | head 00000000 1f 8b 08 00 04 ed 50 4b 00 03 ec 9a 0d 98 55 55 |......PK......UU| 00000010 b9 c7 17 30 0c 87 71 54 b0 d1 48 30 27 43 2f e9 |...0..qT..H0'C/.| 00000020 24 33 c3 08 a8 94 98 a8 a8 63 8d 8a 36 2a 79 38 |$3.......c..6*y8| 00000030 73 ce 9e 39 87 39 5f 9c bd 0f 03 3a 25 16 1a f9 |s..9.9_....:%...| 00000040 51 54 54 68 58 64 98 68 98 9a 68 94 94 dc 61 3e |QTThXd.h..h...a>| 00000050 e8 46 c6 4d 2b 6e 5a 72 6f 53 97 ba 54 52 d4 a5 |.F.M+nZroS..TR..| 00000060 1b 69 bf 77 af 35 b3 5f e6 31 6f 3d 4f fa dc e7 |.i.w.5._.1o=O...| 00000070 b9 9d 87 3f 6b fd f7 bb 3e de f5 be ef fa da 7b |...?k...>......{| 00000080 a6 a7 12 41 62 ba 79 55 7f f5 fc 66 cd 3a c3 a6 |...Ab.yU...f.:..| 00000090 33 6d da d0 58 1f a6 ee 67 1a 1a 66 34 36 35 cc |3m..X...g..f465.|
The file I'm trying to unpack can be downloaded from http://www.konstantin-schauwecker.de/files/joe-3.7-android.tar.gz (http://www.konstantin-schauwecker.de/joe-android.php page)
Btw, if works OK if i first try to unzip it using gzip -d gzip -d joe-3.7-android.tar.gz (both /system/bin/gzip and /system/xbin/gzip works) and when executing tar -xvf joe-3.7-android.tar So there is a problem somewhere in the gzip -> tar conjunction.
(In reply to comment #2) > I have the same(or maybe not the same) problem with decompressing tar.gz files > on Android. I'm using newest BusyBox 1.21.1. > > When I'm trying to unpack the file, it says > > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz > tar -xvZf joe-3.7-android.tar.gz > tar: invalid magic > tar: short read -Z is for auto-decompress, no auto-gunzip > > Please find the hexdump attached > > shell@android:/sdcard/flash $ hexdump -C joe-3.7-android.tar.gz | head > hexdump -C joe-3.7-android.tar.gz | head > 00000000 1f 8b 08 00 04 ed 50 4b 00 03 ec 9a 0d 98 55 55 |......PK......UU| ^^^^^ This file is gzipped, not compressed: COMPRESS_MAGIC = 0x9d1f, GZIP_MAGIC = 0x8b1f,
(In reply to comment #5) > (In reply to comment #2) > > I have the same(or maybe not the same) problem with decompressing tar.gz files > > on Android. I'm using newest BusyBox 1.21.1. > > > > When I'm trying to unpack the file, it says > > > > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz > > tar -xvZf joe-3.7-android.tar.gz > > tar: invalid magic > > tar: short read > > -Z is for auto-decompress, no auto-gunzip So, as i understand, auto g-unzip is just not implemented yet? Is it planned to be implemented?
(In reply to comment #6) > > > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz > > > tar -xvZf joe-3.7-android.tar.gz > > > tar: invalid magic > > > tar: short read > > > > -Z is for auto-decompress, no auto-gunzip > So, as i understand, auto g-unzip is just not implemented yet? Is it planned to > be implemented? It is implemented: $ busybox tar --help BusyBox v1.22.0.git (2013-07-30 11:58:43 CEST) multi-call binary. Usage: tar -[cxtZzJjahmvO] [-X FILE] [-T FILE] [-f TARFILE] [-C DIR] [FILE]... Create, extract, or list files from a tar file Operation: c Create x Extract t List f Name of TARFILE ('-' for stdin/out) C Change to DIR before operation v Verbose Z (De)compress using compress z (De)compress using gzip J (De)compress using xz j (De)compress using bzip2 a (De)compress using lzma O Extract to stdout h Follow symlinks m Don't restore mtime exclude File to exclude X File with names to exclude T File with names to include
Danila, try with tar -xvzf joe-3.7-android.tar.gz instead. tar -xvf joe-3.7-android.tar.gz may work as well
Denys, You closed this PR with RESOLVED/INVALID. I don't see any note why this is an invalid problem. Latest discussion was about gzipped streams, how they are implemented or not. My OP was about unzipped stream. (!!) Please explain why closed this PR. Regards, Yuri
(In reply to comment #7) > (In reply to comment #6) > > > > 1|shell@android:/sdcard/flash $ tar -xvZf joe-3.7-android.tar.gz > > > > tar -xvZf joe-3.7-android.tar.gz > > > > tar: invalid magic > > > > tar: short read > > > > > > -Z is for auto-decompress, no auto-gunzip > > So, as i understand, auto g-unzip is just not implemented yet? Is it planned to > > be implemented? > > It is implemented: > > $ busybox tar --help > BusyBox v1.22.0.git (2013-07-30 11:58:43 CEST) multi-call binary. > > Usage: tar -[cxtZzJjahmvO] [-X FILE] [-T FILE] [-f TARFILE] [-C DIR] [FILE]... > > Create, extract, or list files from a tar file > > Operation: > c Create > x Extract > t List > f Name of TARFILE ('-' for stdin/out) > C Change to DIR before operation > v Verbose > Z (De)compress using compress > z (De)compress using gzip > J (De)compress using xz > j (De)compress using bzip2 > a (De)compress using lzma > O Extract to stdout > h Follow symlinks > m Don't restore mtime > exclude File to exclude > X File with names to exclude > T File with names to include Denis, my version shows that slightly another options: 1|shell@android:/storage/sdcard0/flash $ busybox tar --help busybox tar --help BusyBox v1.21.1-Stericson (2013-07-08 15:58:11 BST) multi-call binary. Usage: tar -[cxtZhmvO] [-X FILE] [-T FILE] [-f TARFILE] [-C DIR] [FILE]... Create, extract, or list files from a tar file Operation: c Create x Extract t List f Name of TARFILE ('-' for stdin/out) C Change to DIR before operation v Verbose Z (De)compress using compress O Extract to stdout h Follow symlinks m Don't restore mtime exclude File to exclude X File with names to exclude T File with names to include Maybe it was included only in 1.22 version?
(In reply to comment #8) > Danila, try with tar -xvzf joe-3.7-android.tar.gz instead. > > tar -xvf joe-3.7-android.tar.gz may work as well It doesn't work with both cases, saying "1|shell@android:/storage/sdcard0/flash $ tar -xvzf joe-3.7-android.tar.gz tar -xvzf joe-3.7-android.tar.gz tar: invalid option -- z" for the first and or that earlier referenced output about invalid magic
(In reply to comment #11) > (In reply to comment #8) > > Danila, try with tar -xvzf joe-3.7-android.tar.gz instead. > > > > tar -xvf joe-3.7-android.tar.gz may work as well > > It doesn't work with both cases, saying > "1|shell@android:/storage/sdcard0/flash $ tar -xvzf joe-3.7-android.tar.gz > tar -xvzf joe-3.7-android.tar.gz > tar: invalid option -- z" for the first > and or that earlier referenced output about invalid magic I've contacted the author of this busybox build, Stephen Erickson, and he have solved the problem. It seems that tar was not build with gzip option enabled.