Since: https://git.busybox.net/busybox/commit/?id=dba0dc1999bb1e8bfe64607e2a9385cda361fcb7 (dd: call fsync() only once before exiting if conv=fsync is specified) Writing a 100MB file using "dd bs=1M conv=fsync" will now only sync once rather than 100 times (once every 1MB), which totally changes the behaviour of dd if the progress of the dd is being monitored by another thread (for example, when updating firmware and attempting to give the user feedback during the dd). Can this change be reverted, or perhaps only perform one fsync if bs is NOT specified? I selected "Standard Compliance" for this component as this new behaviour makes no sense to me. If I wanted to perform only a single sync after a dd, I'd perform a "dd && sync", there's no need to change how conv=fsync works.
In LibreELEC I've reverted this change: https://github.com/LibreELEC/LibreELEC.tv/pull/1862 The reason being that during a system update we dd the new image to the SD card using "bs=1M conv=fsync" and give the user progress feedback during the transfer (ie. a percentage complete indicator, 0-100%) by monitoring the new file size using a second thread. Now, with 1.27.0 and the change to dd conv=fsync, the file size of the new file doesn't change **during** the transfer (as there is no fsync) so the user progress feedback is inaccurate. Instead of a steady progression of 1%, 2%, 3% etc. as data is transferred and synced to the file system, there is now a long delay where there is no progress, then large jumps to 50% and 100% as the data is finally synced at the end of dd. If anyone wanted to achieve what this dba0dc19 change has achieved then they'd simply not use "conv=fsync" at all, and perform a sync after the dd command.
$ dd --version dd (coreutils) 8.25 $ strace -oz dd conv=fsync bs=1M count=2 </dev/zero >zz 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0227291 s, 92.3 MB/s $ tail -20 z open("/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) lseek(0, 0, SEEK_CUR) = 0 mmap(NULL, 1060864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa42fe38000 read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576 read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576 fsync(1) = 0 close(0) = 0 close(1) = 0 write(2, "2+0 records in\n2+0 records out\n", 31) = 31 write(2, "2097152 bytes (2.1 MB, 2.0 MiB) "..., 62) = 62 write(2, "\n", 1) = 1 close(2) = 0 exit_group(0) = ? +++ exited with 0 +++ Well, as seen above, coreutils' dd calls fsync just once, at the end. coreutils dd has "status=progress" thing which prints stats to stderr once per second. Please try whether it is a suitable replacement for your needs. If it is, I can add that to busybox.
Hi Denys. Yes, after opening this issue I did take a look at GNU dd and saw that it does fsync just the once, so I guess that Busybox now matches the behaviour of GNU dd. Unfortunately the status=progress output doesn't appear to be anything we can easily use/parse and reformat into a simplified version for the user. What we need - and have been using successfully up to this point - is a simple 0-100% progress indicator, and one that accurately reflects what is synced to the file system (which would be impossible with only a single fsync). If necessary we'll just continue to revert this change, although I'd argue that changing the behaviour of dd like this (and at this stage for what seems like a marginal benefit) may not be the best idea regardless of how GNU dd behaves - I wouldn't be surprised if this catches out other embedded installations that expect the "old" dd fsync behaviour. It's pretty easy to simulate the GNU dd without this fsync change (dd && sync), but impossible to recreate the old Busybox dd behaviour that fsynced multiple times during the transfer. Would it be possible to keep the old fsync behaviour as the default, but enable this new one-fsync behaviour via a build option? That might be the safest/most compatible solution?
(In reply to Neil MacLeod from comment #3) > If necessary we'll just continue to revert this change, although I'd argue that changing the behaviour of dd like this (and at this stage for what seems like a marginal benefit) may not be the best idea I respectfully disagree. In the long term, importance of not fragmenting Linux world into many slightly incompatible niches is more important than having one or the other incompatible feature which happens to suit some particular need better than "standard" behavior. (I use "standard" loosely here. Basically, for bbox, "standard" is usually what the same-named tool in popular Linux distros is doing). You can petition coreutils to change "fsync", or add a new flag ("fsync_block"?). Then bbox copies that feature. You can find another way to achieve what you want, with other tools/options. For example, maybe running "fsync FILE" (we have this as applet) or "sync" in another process forces writes out, so you can reliably see file size?
OK, no problem. Thanks for considering it anyway.