592

Otherwise, is there any alternative command line utility that can achieve this?

2

24 Answers 24

464

There isn't. See here as to why. Although it does more than you need, rsync has a --progress parameter. The -a will keep permissions,etc, and -h will be human readable.

rsync -ah --progress source destination

The output will look something like this:

Pictures/1.jpg
      2.13M 100%    2.28MB/s    0:00:00 (xfr#5898, to-chk=1/5905)
Pictures/2.jpg
      1.68M 100%    1.76MB/s    0:00:00 (xfr#5899, to-chk=0/5905)
13
  • 22
    This works great in the current Ubuntu (14.10). It also supports the -r flag to recurse directories. It can even be aliased as a direct replacement for cp: alias cp="rsync -ah --progress"
    – rustyx
    Dec 23, 2014 at 21:04
  • 9
    i like this alternative better than pv, specially because rsync is part of the standard install
    – Joao Costa
    Sep 17, 2017 at 18:31
  • 40
    @bartekbrak thanks for sharing. I'm presently on macOS 10.12.x running homebrew version of rsync 3.1.2 and using --progress makes my head spin 🙃 watching all the output display in the terminal. So those coming here from AOL keyword searches, rsync -ah --info=progress2 [source] [destination] presents the output in a little more sane 👩‍⚕️ manner IMHO.
    – ipatch
    Mar 1, 2018 at 19:53
  • 7
    Unfortunately, the output is not very helpful when copying many small files.
    – Raphael
    Apr 28, 2018 at 18:29
  • 22
    @Raphael, true, that's why you should use --info=progress2 rather than --progress as suggested by @ipatch in their comment Jun 19, 2019 at 15:15
406

While cp hasn't got this functionality, you can use pv to do this:

pv my_big_file > backup/my_big_file

Note: this method will lose the file's permissions and ownership. Files copied this way will have the same permissions as if you'd created them yourself and will belong to you.

In this example, pv basically just outputs the file to stdout*, which you redirect to a file using the > operator. Simultaneously, it prints information about the progress to the terminal when you do that.

This is what it looks like:

stefano@ubuntu:~/Data$ pv my_big_file > backup/my_big_file
 138MB 0:00:01 [73.3MB/s] [=================================>] 100% 

You may need to Install pv (alternatively, type sudo apt-get install pv) on your system.


*: The technical bit

There are three important streams of data in a unix-like system: stdout (standard output), stderr (standard error) and stdin (standard input). Every program has all three, so to speak. The > redirection operator redirects program output to a file. Without arguments, as you see above, > redirects a program's standard output to a file. cp basically does nothing fancier than

cat source > destination

(where cat just reads a file and prints it to stdout). pv is just like cat, but if you redirect it's output stream somewhere else, it will print progress information to stdout instead.

Take a look at man pv to learn more about it.


Another option, as DoR suggests in this answer, is to use rsync instead:

$ rsync -ah --progress source-file destination-file
sending incremental file list
source-file
        621.22M  57%  283.86MB/s    0:00:01

This will preserve the files permissions/ownership while showing progress.

16
  • 3
    It may or may no be significant (depending on the situation), but pv does not handle permissions the same way as cp does... (based on one quick test I tried: pv didn't copy the execute bit across.. rsync did.
    – Peter.O
    Dec 14, 2010 at 15:55
  • 70
    IMO: alias cp="rsync -avz" cp is outdated. Dec 26, 2010 at 1:26
  • 11
    If you're like me, and forget about pv, you can go snooping in /proc/PID of cp/fd and /proc/PID of cp/fdinfo to figure out progress. (It's up to you to infer speed.) I use this technique to watch updatedb.
    – Thanatos
    Dec 26, 2010 at 1:30
  • 21
    Yes, -z should probably only be used for network copies; compressing and decompressing the data for a local copy is pure overhead. Jun 17, 2015 at 15:17
  • 5
    @MarcoCeppi , when copying directory with rsync be sure not to add trailing / to source path (or remove if e.g. bash completion put it there automatically). Otherwise you will get results different than when using cp (or gcp). Apr 3, 2016 at 19:07
119

I get a kick out of using cURL for this exact purpose. The man page lists the "FILE" protocol as supported, so just use it like any other protocol in a URL:

curl -o destination FILE://source

Speed, progress, time remaining, and more -- all in a familiar format.

9
  • 4
    This is great especially in environments where you are prohibited from installing new tools and where rsync is unavailable.
    – user190264
    Sep 5, 2013 at 7:00
  • 19
    Brilliant, this has got to be a must-known hack!
    – ionreflex
    Oct 20, 2015 at 18:13
  • 5
    This presumably has the same drawback as pv, that it won't save permissions.
    – user435587
    Jul 9, 2018 at 0:09
  • 3
    Sweet! any way to make this work for a directory?
    – Joe M
    Feb 25, 2019 at 2:21
  • 3
    -O (capital O) to use the source filename instead of having to write destination
    – Codebling
    Oct 10, 2020 at 4:12
115

If you want to see if your files are transferring correctly you could use gcp and gcp is like cp but by default gives you a progress bar so that you can see what is being copied. As the program's wiki notes, gcp has several useful features such as

  • transfer progression indication
  • continuous copying on error (skip to next file)
  • copy status logging: gcp logs all its actions so that it is possible to know which files have been successfully copied
  • name mangling to handle target filesystem limitations (for example deletion of incompatible characters "*" or "?" on FAT)

However, even when the progress bar has reached 100% when using the tool, you must wait until your terminal prompt reappears before safely removing your media so that you can ensure that the transfer process has successfully finished.

gcp is used to copy files and has options such as --preserve so that various attributes and permissions can be preserved and --recursive so that whole directories can be copied. More information on its options can be found by entering man gcp or by going to the Ubuntu manpages online. A tutorial is also available on this site.

Install gcp from the repositories with

sudo apt-get install gcp

(Note: in Ubuntu 12.10 the new automount point is, for example, /media/user/usbdisk)

You can copy a file to your media by entering

gcp /home/mike/file.mp4 /media/usb

and copy a folder to your media with

gcp -rv ~/Podcasts /media/Mik2

Sample output from gcp with the progress bar:

gcp ~/Videos_incIplayer/mars.flv /media/Mik2
Copying 168.57 MiB 100% |########################################################|   7.98 M/s Time: 00:00:22

You can of course specify multiple files or folders to copy to your disk, and there are a lot of other options covered in man gcp.

5
  • 7
    Got error with it over ssh dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NotSupported: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11 Jan 17, 2016 at 23:29
  • 3
    if you have a X11 display open on the machine you can just set export DISPLAY=:0.0 before starting gcp. If the machine is headless then you'd have to start a xsession into a virtual framebuffer or something, at that point you should probably just look for another program
    – PeterT
    Oct 9, 2016 at 17:33
  • 3
    gcp is pretty decent but the DBUS/X requirement is odd. I handled this over SSH by using ssh forwarding: ssh -X user@host and that allowed me to run it.
    – Oli
    Oct 21, 2016 at 12:24
  • 1
    Got error /usr/bin/gcp:42: DeprecationWarning: Importing dbus.glib to use the GLib main loop with dbus-python is deprecated. Instead, use this sequence: from dbus.mainloop.glib import DBusGMainLoop DBusGMainLoop(set_as_default=True)
    – Marecky
    May 23, 2021 at 15:11
  • 2
    I cannot recommend gcp, 1. it is not maintained. 2. it cannot handle symlinks, hardlinks, etc. 3. it cannot handle sparse file.
    – Wang
    Dec 23, 2022 at 18:52
66

There is a tool called progress in the repositories that is able to examine various different commands and display progress info for them.

Install it using the command

sudo apt-get install progress

This tool can be used like that:

cp bigfile newfile & progress -mp $!

Output:

[11471] cp /media/Backup/Downloads/FILENAME.file 
        29.9% (24.2 MiB / 16 MiB)
3
  • 5
    This is the most up to date answer in my opinion in 2019.
    – mchid
    Oct 18, 2019 at 4:00
  • 2
    progress is not useful for general scripting, only in a context of watch: a) It needs a fullscreen tty and fails with Error opening terminal: unknown. else. Hence you cannot report progress to a pipe to display progress in a CGI environment. b) Modern shells do not let live zombies long enough, freeing the PID for other processes. Hence there is a possible race condition where progress could monitor the wrong process (i. E. progress is loaded via NFS). c) It is unable to monitor things like sleep which is possible with strace or dtrace and a bit of additional scripting.
    – Tino
    Dec 15, 2020 at 14:10
  • 1
    Working in 2021 like a charm, thank you so much Jul 21, 2021 at 7:23
40

While it doesn't display speed when copying multiple files, the -v option to the cp command will provide you with progress info. e.g.

    cp -rv old-directory new-directory
2
  • 16
    Progress info? This is just verbose output. To provide progress info you would at least need to now how many files, or even which files, needs to be copied. Jul 3, 2015 at 20:21
  • 5
    It shows what file it is on, which can be enough in less demanding cases. Apr 27, 2019 at 23:54
21

The kernel knows most of the data such as speed, and often also percentage. Modern kernels expose this via their /proc filesystem.

showspeed from https://github.com/jnweiger/showspeed uses that info. It can attach to already running programs and give periodic updates like this:

$ dd if=bigfile of=/tmp/otherbigfile &
$ showspeed dd
dd looks like a process name. pid=4417 matches av0=dd.
p/4417/fd/0r /home/jw/bigfile 113MB/s (12%, 2.3GB)  9m:35
p/4417/fd/1w /tmp/otherbigfile 182MB/s (2.6GB)
p/4417/fd/0r /home/jw/bigfile 285MB/s (15%, 3.0GB)  8m:08
p/4417/fd/0r /home/jw/bigfile 115MB/s (16%, 3.2GB)  8m:01
p/4417/fd/0r /home/jw/bigfile 107MB/s (17%, 3.4GB)  7m:39
p/4417/fd/1w /tmp/otherbigfile 104MB/s (3.5GB)
p/4417/fd/0r /home/jw/bigfile 139MB/s (19%, 3.7GB)  7m:37
p/4417/fd/0r /home/jw/bigfile 116MB/s (20%, 3.9GB)  7m:18
p/4417/fd/1w /tmp/otherbigfile  67MB/s (4.0GB)
p/4417/fd/1w /tmp/otherbigfile 100MB/s (4.1GB)
...
3
  • Nice one. How does it work out percentages though? I guess only in case of file s open as read only? And it will probably not work in case of scattered access.
    – j_kubik
    Dec 22, 2014 at 17:29
  • 1
    In this case you can also use another terminal to run pkill -USR1 dd to make dd show its status, another option would be watch -n 1 pkill -USR1 dd to make it show its progress periodically (every second).
    – Yaron
    Mar 16, 2015 at 9:23
  • Right. Many tools got a builtin way to report statistics over time. dd is no exception. Modern implementations have a status=progess option. Showspeed is a classic unix style "one tool for one purpose" solution - just like pv. But it has different usecases: Think of cups pumping a file through ghostscript or you want to know the ETA for an ongoing cp or tar. It might be 95% done already after a few hours and you probably don't want to restart that just to add pv ... Scattered access would not work well. Showspeed only samples seek positions. Apr 30, 2018 at 19:22
14

While pv can deal with local cp tasks, using dd with pv can deal with both local (cp) and remote (scp) tasks.

dd if=path/to/source.mkv | pv | dd of=path/to/dest.mkv

Please ensure the path/to/dest.mkv exits by touch path/to/dest.mkv

This can show the progress, but if you want the percentage information,

dd if=path/to/source.mkv | pv -s 100M | dd of=path/to/dest.mkv

Replace 100M above with the real size of your source file.

Here Comes the Remote Part

While scp can hardly show current progress, using dd with pv is a piece of cake.

ssh onemach@myotherhost dd if=path/to/source.mkv | pv -s 100M | dd of=path/to/dest.mkv
1
13

dd status=progress

Option added in GNU Coreutils 8.24+ (Ubuntu 16.04):

dd if=src of=dst status=progress

The terminal shows a line of type:

462858752 bytes (463 MB, 441 MiB) copied, 38 s, 12,2 MB/s

that gets regularly updated.

See also: How do you monitor the progress of dd?

3
  • 1
    You meant: stdout is a stream of characters and vt100 escape sequences that make your terminal display a periodically updated line ;). stdout is always a stream of bytes and has nothing to do with how it is displayed (except that the application can know whether it is currently attached to a terminal or not...)
    – masterxilo
    Sep 4, 2018 at 9:20
  • @masterxilo newbs won't care, experts will know ;-) hehe Sep 4, 2018 at 9:33
  • Well I think it's always good to state things like they are, maybe leaving out some details, but never make your readers produce a wrong mental model. If you say "stdout is a (periodically updated) line" it seems like you are saying stdout is a block of mutable memory which it is not. This is important to know even for newbies.
    – masterxilo
    Sep 7, 2018 at 9:09
12

If you have rsync 3.1 or higher (rsync --version), you can copy (cp -Rpn) while preserving permissions and ownership, recurse directories, "no clobber," and display overall progress (instead of just progress by file), copy rate, and (very rough) estimated time remaining with:

sudo rsync -a --info=progress2 --no-i-r /source /destination

Note that sudo is only needed if dealing with directories/files you don't own. Also, without the --no-i-r, the percentage may reset to a lower number at some point during the copy. Perhaps later versions of rsync will default to no-i-r with info=progress2, but it does not in the current version of 3.1.2.

I've found that the percentage and time remaining are grossly overestimated when copying to a directory that already contains files (ie. like when you would typically use cp -n "no clobber").

0
11

There's a new tool called cv that can find any descriptor related to a running command and show progress and speed: https://github.com/Xfennec/cv

cv -w

outputs the stats for all running cp,mv etc. operations

3
  • watch cv -q is even neater
    – nwgat
    Jan 19, 2015 at 0:42
  • It only show progress for an individual files, it does realise whether a cp is copying a directory recursively or not.
    – Flimm
    Mar 31, 2016 at 9:27
  • 2
    It war renamed to progress. Great tool!
    – sebastian
    Jun 25, 2016 at 14:42
10

There are already some really great answers here, but here are my favorite commands, including my own little spin on the rsync command, with my favorite options:

1. watch 'du -hs' (or, for more granularity: watch -n 1 'du -sk | awk '\''{printf "%.3f MiB %s\n", $1/1024, $2}'\''') allows you to watch a destination folder grow in size if cp is already running:

What if you already started a long cp process and don't want to stop it now?

Assuming you already started this copy command:

# recursively copy source_dir_name to destination
cp -r source_dir_name destination

Just open up a new terminal, with the copy operation already in progress, and run:

# cd into destination directory
cd path/to/destination/source_dir_name

# Now watch, live, the size of this destination grow!
watch 'du -hs'

# OR (Preferred!) see even more granularity, in X.xxx MiB: 
watch 'du -sk | awk '\''{printf "%.3f MiB %s\n", $1/1024, $2}'\''' 

This will cause the watch command to automatically run du -hs (or du -sk | awk '{printf "%.3f MiB %s\n", $1/1024, $2}') every 2 seconds (the default for watch) inside your destination directory, so you can see the size of the folder grow in real-time! This doesn't give you a percentage complete, but if you have a rough idea of the size of what you're copying, it certainly gives you something to watch and feel good about, knowing the destination is growing. Here's a sample output on the watch screen:

  1. Cmd: watch 'du -hs'

    Output:

    Every 2.0s: du -hs
    
    2.5G    .
    
  2. Cmd: watch 'du -sk | awk '\''{printf "%.3f MiB %s\n", $1/1024, $2}'\'''

    Output:

    Every 2.0s: du -sk | awk '{printf "%.3f MiB %s\n", $1/1024, $2}'
    
    2560.348 MiB .
    

Above, you can see the current size of the current directory (.) is 2.5 GiB, or, more specifically in the 2nd case: 2560.348 MiB. To make watch use a slower update interval, such as 5 seconds, just add -n 5 to the watch command, like this:

watch -n 5 'du -hs'

# OR (preferred, for more granularity)
watch -n 5 'du -sk | awk '\''{printf "%.3f MiB %s\n", $1/1024, $2}'\'''

2. rsync really is best for this:

My favorite commands:

# dry-run
time rsync -rah --dry-run --info=progress2 --stats source destination

# real copy (Important! Do a dry run first and read the
# output summary to ensure you're copying what you intend!)
time rsync -rah --info=progress2 --stats source destination

Note: the time part at the front of the above commands just outputs a nice time summary of the total run-time for the operation in the end. This works before any Linux command.

Here's what an rsync --stats summary for a --dry-run looks like, by the way:

$ rsync -rah --dry-run --info=progress2 --stats /media/gabriel/cdrom ~/temp
          2.76G 100% 2570.10GB/s    0:00:00 (xfr#3836, to-chk=0/4837)   

Number of files: 4,837 (reg: 3,836, dir: 1,001)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 3,836
Total file size: 2.76G bytes
Total transferred file size: 2.76G bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 123.88K
Total bytes received: 15.54K

sent 123.88K bytes  received 15.54K bytes  278.85K bytes/sec
total size is 2.76G  speedup is 19,792.78 (DRY RUN)

References:

  1. Place where I first documented the watch -n 5 'du -sk | awk '\''{printf "%.3f MiB %s\n", $1/1024, $2}'\''' type command, in my ROS tutorial: my ROS.org tutorial: Reading messages from a bag file

Extra reading:

  1. My long answer on some advanced rsync usage and stuff: https://superuser.com/questions/1271882/convert-ntfs-partition-to-ext4-how-to-copy-the-data/1464264#1464264
  2. [another one of my answers] Unix & Linux: Is it possible to see cp speed and percent copied?
9

As many said, cp does not include this functionality.

Just to throw my $0.02, what I usually do with trivial copying situations (i.e. no -R):

  1. See how big the file is and remember

  2. Start copying

  3. Open another terminal

  4. Run watch ls -lh DIR on the directory where the target is

This can keep me updated on target file size, with quite a minimum hassle.

As an alternative for less trivial situations, e.g. recursively copying directories, you can use watch du -hs DIR to see summary of DIR size. However du can take long to compute and can even slow down the copying, so you might want to use -n INTERVAL argument to watch so that trade-off is acceptable.

Update: In case you use wild-cards with command used with watch du, e.g. watch du -hs backup/*, don't forget to quote:

watch "du -hs backup/*"

otherwise the wild-cards will be expanded only once, when watch is started so du will not look at new files / subdirectories.

1
  • Does not answer the "speed" part, though... Nov 16, 2012 at 13:21
5

pv knows how to watch file descriptors given a pid, whether it's cp or something else

From the documentation:

   (Linux only): Watching file descriptor 3 opened by another process 1234:

          pv -d 1234:3

   (Linux only): Watching all file descriptors used by process 1234:

          pv -d 1234

Example:

md5sum file &
[1] + 1271 suspended
pv -d 1271
417MiB 0:00:17 [12,1MiB/s] [============>                                 ] 29% ETA 0:00:53

$ cp file.mov copy.mov &
[2] 3731
$ pv -d 3731
3:/media/windows/file.mov:  754MiB 0:00:04 [97,2MiB/s] [======================>                       ] 52% ETA 0:00:07
4:/media/windows/copy.mov:  754MiB 0:00:04 [97,3MiB/s] [     <=>                                                                   ] 
1
  • This should be incorporated into the selected answer. Feb 5, 2020 at 13:07
4

Use a shell script:

#!/bin/sh
cp_p()
{
   strace -q -ewrite cp -- "${1}" "${2}" 2>&1 \
      | awk '{
        count += $NF
            if (count % 10 == 0) {
               percent = count / total_size * 100
               printf "%3d%% [", percent
               for (i=0;i<=percent;i++)
                  printf "="
               printf ">"
               for (i=percent;i<100;i++)
                  printf " "
               printf "]\r"
            }
         }
         END { print "" }' total_size=$(stat -c '%s' "${1}") count=0
}

This will look like:

% cp_p /home/echox/foo.dat /home/echox/bar.dat
66% [===============================>                      ]

Source

4

Depending on what you want to do, Midnight Commander (mc) might be the answer. I'm surprised it's not been mentioned yet.

Tools like pv or rsync are good to display progress of transfer of one huge file, but when it comes to copying whole directories/trees, mc calculates the size and then displays the progress very nicely. Plus it's available out of the box on majority of systems.

1
  • 3
    "it's available out of the box on majority of systems." ... not on any default Ubuntu installation.
    – muru
    Jan 30, 2016 at 17:58
2

one more option to preserve attributes could be (if source is a folder it will be created in destination)

tar -c source | pv -e -t -p -r | tar -C destination  -x

hope it may be useful to someone. To have estimated transfer time this can be acheived by doing do -s source in advance and passing it as a -s <size> parameter to pv.

4
  • -f - is redundant. tar's default output for c and input for x are stdout and stdin.
    – muru
    Oct 17, 2018 at 16:34
  • May it depend on the OS/tar version? I know it’s an Ubuntu site yet this may be useful for other OS owners like macOS
    – ciekawy
    Oct 17, 2018 at 16:39
  • 1
    Nope. All sane implementations (including libarchive's tar, which macOS uses) have this behaviour.
    – muru
    Oct 17, 2018 at 16:46
  • just verified on macos and your suggestion is perfectly valid - I've updated my answer. Thanks!
    – ciekawy
    Oct 17, 2018 at 17:21
2

You can monitor your progress using watch

cp -r /path/to/your/file/or/dir ./dst

and in second terminal you can watch your progress

watch -n 0.1 du -h --max-depth=1
1
  • This solution works the best for me! Just in case someone wants to know, the second command needs to be run from inside the destination folder. Dec 13, 2022 at 0:08
1

I use xcp to get a fancy progress bar:

xcp -r myfolder /path/to/dest/folder

If you already have rust installed, install it with cargo install xcp.

0

You can copy use any program. At the same time, you can start sudo iotop and see the actually disk read/write speed yet without progress.

1
  • Yep, but the OP explicitly asked for a progress bar, so that would be needed...
    – Mayou36
    Apr 29, 2022 at 10:56
0

Check the source code for progress_bar in the below git repository

https://github.com/Kiran-Bose/supreme

Also try the custom bash script package supreme

Download the deb file and install in debian based distribution or download the source files, modify and use for other distros

Functionality overview

(1)Open Apps ----Firefox ----Calculator ----Settings

(2)Manage Files ----Search ----Navigate ----Quick access

            |----Select File(s)
            |----Inverse Selection
            |----Make directory
            |----Make file
                                          |----Open
                                          |----Copy
                                          |----Move
                                          |----Delete
                                          |----Rename
                                          |----Send to Device
                                          |----Properties

(3)Manage Phone ----Move/Copy from phone ----Move/Copy to phone ----Sync folders

(4)Manage USB ----Move/Copy from USB ----Move/Copy to USB

0

As linux original cp has no this functionaliy, I used the following solution to get the progress information:

cp -a source_directory destination_direcotry

During the copy is working, create another command terminal and go to the source_directory as:

cd source_directory
du -sm .

After I got the total size of the source_directory, then go to the destination_direcotry to do the same work:

cd destination_direcotry
du -sm .

I could get the copying progress by comparining the total size difference between the destination_direcotry and the source_directory, the only extra work you need do is opening another terminal besides the cp terminal, to type "du -sm ." to get the total directory size you are coping.

0

Terminal screenshot

Install gcp

sudo apt-get install gcp

NOTE: Make Sure you have enable X11 support if not: Add following line to '/etc/ssh/sshd_config'

ForwardX11Trusted yes

Install

sudo apt-get install xauth
reboot

When its done:

gcp -rv SOURCE_FOLDER DESTINATION_FOLDER

or:

gcp SOURCE_FILE DESTINATION_FILE
1
  • My sshd already had this option and xauth was installed, I get the error anyway.
    – bomben
    Oct 31, 2022 at 8:45
0

Not a perfect option you can open a second terminal and do

while true; do ls -lh [location/filename]; sleep 2; clear; done

It will show you the file in human readable format and you can watch the size of the file grow larger. I do this when I'm not able to install new utilities on the machine.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .