Bug 6908

(In reply to comment #0)
> I have a directory on Android which has one file of 3+GB and the other three
> files that are much smaller.
> 
> I am trying to transfer the directory from the Android (from shell) to the BSD
> box. For this purpose I run this command:
> tar cf - my-directory | ssh user@host "cd /somedir && tar xf -"

What "tar --help" says? (Is it bbox's tar? Which version?)

> What I observe:
> Command succeeds, nothing is printed.
> Only two files are left on the BSD box in /somedir/my-directory
> One smaller file went first and is transferred correctly.
> 3+GB file has the right size, but the contents are damaged (md5 hash mismatch)
> Two other files failed to transfer.
> 
> 3+GB file on destination grew gradually to ~1.3GB, after which it quickly grew
> to the right size. hexdump shows that after 1.3GB the file contains only zeros.

Can you run " tar cf - my-directory | hexdump -C | tail -n100000"
and show the (part of) the output where file data ends and zeros start?
(The "part of" should be large - NOT three lines! more like 300 or more
lines around the problematic part)

> What "tar --help" says? (Is it bbox's tar? Which version?)

BusyBox v1.21.0 (2013-07-08 16:00:47 CEST) multi-call binary.


> Can you run " tar cf - my-directory |  "
> and show the (part of) the output where file data ends and zeros start?
> (The "part of" should be large - NOT three lines! more like 300 or more
> lines around the problematic part)

Now I created a new large file huge/huge3plus with random data:
# ls -l huge/                                                                                          
----rwxr-x    1 system   sdcard_r 3815000000 Feb 21 12:09 huge3plus

Trying to create an archive:
# tar cf - huge > huge.tar

Resulting tar archive is obviously too small:
# ls -l huge.tar
----rwxr-x    1 system   sdcard_r 479969408 Feb 21 12:10 huge.tar

Running suggested by you command:
# cat huge.tar | hexdump -C | tail -n100000 > huge.tar.last1k.hex

Please see the compressed huge.tar.last1k.hex attached.

Created attachment 5246 [details]
tail of the hex dump of the damaged tar archive

Due to the attachment size limitations, I had to truncate to 2000 last lines. Hope this is still useful.
I can upload the whole huge.tar if this would help.

(In reply to comment #3)
> Created attachment 5246 [details]
> tail of the hex dump of the damaged tar archive
> 
> Due to the attachment size limitations, I had to truncate to 2000 last lines.
> Hope this is still useful.

Hmm. I don't see anything obviously broken there.

It doesn't look like a case of "modulo 2Gb" bug: the file you are compressing is 0xe36447c0 bytes,
the tarball is only 0x1c9bc080 bytes, nowhere near 0x636447c0.
It's also strange that tarball size isn't a multiple of 512 bytes!

> I can upload the whole huge.tar if this would help.

You can do that indeed, but it will be much smaller if your source file is easily compressible instead of random. (File contents is not likely to be the cause of the bug). Re-create it with the command

$ yes 1234567890abcde | dd bs=16 count=$((3815000000/16)) >huge

so that the file is a repeating sequence of 16 bytes "1234567890abcde\n",
then tar it up, bzip2 it (it will be small), and attach the file here.

I did what you suggested. Resulting tar archive was again exactly the same size. bz2 archive is 49531 bytes, larger than the attachment limit of 35k. I am e-mailing it to you.

Created attachment 5252 [details]
strace of the tar command

Attaching the requested output of this command:
(strace -s99 busybox tar cf - huge3plus >/dev/null) 2>&1 | bzip2 >z.bz2

(In reply to comment #6)
> Created attachment 5252 [details]
> strace of the tar command
> 
> Attaching the requested output of this command:
> (strace -s99 busybox tar cf - huge3plus >/dev/null) 2>&1 | bzip2 >z.bz2

Thanks. strace shows 117181 reads of 4096 bytes from the file, then one final read of 2112 bytes and then tar writes EOF marker and exits:

...
read(3, "1234567890abcde\n1234567890abcde\n1234567890abcde\n1234567890abcde\n1234567890abcde\n1234567890abcde\n123"..., 2112) = 2112
write(1, "1234567890abcde\n1234567890abcde\n1234567890abcde\n1234567890abcde\n1234567890abcde\n1234567890abcde\n123"..., 2112) = 2112
close(3)                                = 0
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 64) = 64
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
close(1)                                = 0
mprotect(0x4018a000, 4096, PROT_READ|PROT_WRITE) = 0
mprotect(0x4018a000, 4096, PROT_READ)   = 0
munmap(0x4018a000, 4096)                = 0
exit_group(0)                           = ?

In total, tar reads 479975488 bytes (~457 Mb) which matches tarball size. Note that tarball did not reach EOF on read. This is because it uses stat() result to read exactly as many bytes as stat() indicated:

                /* We record size into header first, */
                /* and then write out file. If file shrinks in between, */
                /* tar will be corrupted. So we don't allow for that. */
                /* NB: GNU tar 1.16 warns and pads with zeroes */
                /* or even seeks back and updates header */
                bb_copyfd_exact_size(inputFileFd, tbInfo->tarFd, statbuf->st_size);

But stat() did give us the correct size:

lstat64("huge3plus", {st_mode=S_IFREG|075, st_size=3815000000, ...}) = 0
open("huge3plus", O_RDONLY)             = 3

bb_copyfd_exact_size() has a special convention for negative size
parameter:
"size < 0 means "ignore write errors", used by tar --to-command".
If 3815000000 is treated as negative (IOW: if your libc have 32-bit off_t),
then it will think that it was given size = -479967296. Hmm. Looks like that's it.

Do you have CONFIG_LFS=y in your .config?
What version of libc do you use?
What is its sizeof(off_t)?

Thank you for your help!

So 'stat' returns the correct size, but sequential reads hit the end of file? I never saw this on unchanged in the process file. This should only happen if someone outside truncated the file in the process (which isn't our case).

This is an Android system installed by CyanogenMod (http://www.cyanogenmod.org) on a (rooted) cellphone. I didn't build it myself, this is their stock build
cm_quincyatt-userdebug 4.2.2 JDQ39E 9a899040f2

> Do you have CONFIG_LFS=y in your .config?
No such file exists in the file system. There is no /etc/make.conf either

> What version of libc do you use?
Android doesn't use the usual gnu glibc library. They use some trimmed down version of netbsd libc. I am not sure how to get the version on Android.

> What is its sizeof(off_t)?
Android doesn't come with the compiler installed. Do you know some way to learn this using some shell command? (I doubt it exists though)


I posted this question on CyanogenMod forums: http://forum.cyanogenmod.com/topic/88952-file-reading-is-broken-on-the-large-files-3gb Maybe somebody there can explain what is going on.
I could never suspect that this would be a problem in the base system. It might even be a general Android bug.

(In reply to comment #8)
> Thank you for your help!
> 
> So 'stat' returns the correct size, but sequential reads hit the end of file? I
> never saw this on unchanged in the process file. This should only happen if
> someone outside truncated the file in the process (which isn't our case).

No, it's bionic fk-up with off_t being 32-bit while other libc's support
transparent redefining it as 64-bit.

In bionic, stat->st_size is not off_t, but rather loff_t (64-bit).

Busybox code so far assumed that if it builds with CONFIG_LFS=y,
then off_t is large integer (so far always 64-bit).

I will use this bug as a tracker for that problem.

*** Bug 5096 has been marked as a duplicate of this bug. ***

Another manifestation of this problem:

strace -f /bin/df
...
statfs("/proc/fs/nfsd", {f_type=0x6e667364, f_bsize=4096, f_blocks=0,
f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=0}) = 0
statfs("/var/media/autofs", {f_type=0x187, f_bsize=4096, f_blocks=0, f_bfree=0,
f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=0}) = 0
statfs("/var/media/autofs/GMX", 0x7fb59f28) = -1 EOVERFLOW (Value too large for
defined data type)

We must use statfs64 here.

So it looks like this is the problem in tar arithmetic due to this off_t/loff_t issue.

Also, it looks like Busybox tar memorizes the size in the beginning of the file reading, and then expects the size to be the same in the end. However, file can change in the middle of the process. I think the wiser approach would be if tar would keep reading the file as if it didn't know its size, and if the de-facto size (based on incomplete read call) is different, tar would issue a warning ("tar: The file 'filename' size has changed during archiving from the size NN1 bytes to the size NN2 bytes").
Assumption of the file not changing might cause some other random failures in tar when the file actually got truncated by the outside process.

Also, tar opens file like this:
open("huge3plus", O_RDONLY)             = 3
Isn't it supposed to open it with O_LARGEFILE? In order to read arbitrary files one needs to always set this flag on Linux, or use open64.

(In reply to comment #12)
> So it looks like this is the problem in tar arithmetic due to this off_t/loff_t
> issue.

Other libc handle it correctly: "man fstat"

           struct stat {
...
               off_t     st_size;    /* total size, in bytes */
...
           };

See? st_size must be off_t. Bionic is doing it wrong.

> Also, it looks like Busybox tar memorizes the size in the beginning of the file
> reading, and then expects the size to be the same in the end. However, file can
> change in the middle of the process.

tar file records file size in the header.
If file shrinks mid-flight, bbox tar aborts rather than generating
invalid tar archive.

> I think the wiser approach would be if tar
> would keep reading the file as if it didn't know its size, and if the de-facto
> size (based on incomplete read call) is different, tar would issue a warning
> ("tar: The file 'filename' size has changed during archiving from the size NN1
> bytes to the size NN2 bytes").
> Assumption of the file not changing might cause some other random failures in
> tar when the file actually got truncated by the outside process.

See above.

> Also, tar opens file like this:
> open("huge3plus", O_RDONLY)             = 3
> Isn't it supposed to open it with O_LARGEFILE? In order to read arbitrary files
> one needs to always set this flag on Linux, or use open64.

That's bionic fault. On glibc, uclibc and musl, O_LARGEFILE gets passed into open.
Bionic devels decided that 32-bits will be enough for anyone?

yes, bionic has bad LFS support for existing 32bit ports.  there's not a lot we can do here.