Making nautilus/cp/scp/rsync “progress” cache-aware

Question

When transferring moderately large files (~500-800 MB) to/from a USB disk in nautilus, or using (s)cp (with progress bar from http://chris-lamb.co.uk/2008/01/24/can-you-get-cp-to-give-a-progress-bar-like-wget/), or when using rsync... the progress reported is far from accurate. Evidently the default file system cache size in ubuntu 11.10 64bit is quite a bit larger than for 32 bit. As a consequence, it appears to nautilus or rsync or whatever is doing the copying that the file has been completely copied, when in fact there is a large portion of the file left to write, but that entire portion is has been cached, so it appears to the program that the file has been written (i.e. because it has, from the perspective of system calls). However, the actual output file takes a good 10-20 seconds more to actually fill up. Is there anyway to tweak any of these pieces of software to report progress in terms of "how much data has been written to the actual output file" instead of "how much data has been written to the operating system".

My particular annoyance is that each of these programs reports that the file has been written, but then just hangs for a while reporting nothing. I suspect this is because the "write" system call has completed, but the "close file" system call is waiting for the cache to flush.

Answer 1

I use ubuntu 11.10 64-bit as well. You can use lsof command to poll and see if the cache has been written. This command lists which processes/commands keep files open. I'm not sure if it will work for usb stick cache.

• I tested with rsync a ~2gb file transfer
• usb stick: Bus 002 Device 004: ID 8564:1000 (Transcend Jetflash 8gb retractable, one FAT32 partition)

• mount command shows:

  /dev/sdd1 on /media/19D1-8868 type vfat (rw,nosuid,nodev,uid=1001,gid=1001,shortname=mixed,dmask=0077,utf8=1,showexec,flush,uhelper=udisks)
  

• Command used:

  rsync -Pr ./testfile /media/19D1-8868/test/
  

"-P" shows progress.

lsof while transferring:

$ lsof +f -- /media/19D1-8868/
COMMAND   PID   USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
rsync   23132 testme  cwd    DIR   8,49      4096   33 /media/19D1-8868/test
rsync   23133 testme  cwd    DIR   8,49      4096   33 /media/19D1-8868/test
rsync   23133 testme    1u   REG   8,49 666271744  350 /media/19D1-8868/test/.testfile.2Ejgei

lsof when done:

$ lsof +f -- /media/19D1-8868/

However, I cannot reproduce what you say with rsync. I could almost instantly (2-3 seconds) remove the usb using "Safely remove" option.

Also, when I tried transferring with nautilus, the transfer got stuck to "2 seconds left", until it transferred the whole file, probably including cache because the usb led light stopped blinking and I could again almost instantly safely remove the stick.

1. What does mount show in your case?
2. Did you try to boot from a live cd and test it there too?
3. Did you try to completely format the usb stick with gparted and create a new fat32 partition?