12

Background

In Linux you can:

  • List a group of files with ls Filename*
  • Remove a group of files with rm Filename*
  • Move a group of files with mv Filename* /New/Directory
  • But you cannot copy a group of files with: cp Filename* *.bak

Change Linux cp command to copy group of files

I have a group of files I'd like to copy without entering the names one by one and using the cp command:

$ ls gmail-meta3*
gmail-meta3                          gmail-meta3-REC-1558392194-26467821
gmail-meta3-LAB-1558392194-26467821  gmail-meta3-YAD-1558392194-26467821

How can I use something like the old DOS command copy gmail-meta3* *.bak?

I don't want to type similar command four times:

cp gmail-meta3-LAB-1558392194-26467821 gmail-meta3-LAB-1558392194-26467821.bak

I'm looking for a script/function/app that accepts parameters for old and new filename group and not something with hard-coded filenames. For example, a user can type:

copy gmail-meta3* *.bak

or they might type:

copy gmail-meta3* save-*
  • 1
    The issue looks to me using glob operator twice which none of your other commands use. bash isn't smart enough to handle it. – qwr Jun 26 at 13:32
  • 2
    @qwr, the fact that bash expands metacharacters and tokenizes the input prior to handing it off to a command to be exec'ed is part of the design of the UNIX shells. To somehow try and program an exception for the cp command would break the entire consistency of bash, which would not be smart at all. As an exercise, try and figure out what is happening here, and why it is the shell's metacharacter expansion that makes it so: touch aa ab ba; mkdir bb; cp a* b*; ls * – Mike S Jun 27 at 2:11
  • @MikeS Thanks for the pointers. Yesterday someone else said you can use a wildcard * for the source filenames but not for the target filenames. As such as substitute wildcard (I think ## was suggested but I'm leaning towards %) will have to be used for the target. I think this is what you are reinforcing? I didn't expect to change the cp command at all. Just create a wrapper script called copy that emulated (within reason) the DOS copy command. – WinEunuuchs2Unix Jun 27 at 2:27
  • @WinEunuuchs2Unix that person was correct. The shell metacharacters are independent of the commands. So all wildcards will try and match all files that match the pattern. If you're trying to make a general-purpose "match everything and copy them to whatever they were but add this suffix" program, then yes, putting an unescaped metacharacter as the target will probably not do what you want. Because all metacharacters in the shell command line are expanded. If you KNOW for certain that the target metacharacter will never form a match, you could use it- because the shell can't expand it. – Mike S Jun 27 at 2:56
  • ...but that would be an ugly thing to do. Better to use a special character. % or underscore are good, they're generally not metacharacters (but be careful of using % in a crontab file; it's special there). – Mike S Jun 27 at 2:57
13

Here is an example of one atypical usage of sed, that is applicable for this task:

sed -i.bak '' file-prefix*

In this way, actually, sed will not change the files, because we didn't provided any commands '', but due to the option -i[suffix] it will create a backup copy of each file. I found this approach when I was searching Is there any way to create backup copy of a file, without type its name twice?

  • FYI: $ time sed -i.bak '' gmail-meta3* = real 0m0.069s – WinEunuuchs2Unix Jun 26 at 23:12
  • If files already exists then: real 0m0.037s. If files deleted and run a second time close to cp speed: real 0m0.051s. – WinEunuuchs2Unix Jun 26 at 23:22
  • @xiota An interesting point. Turns out sed is faster when target files already exist but cp is slower when target files exist. I actually flush caches and buffers rather than sync when doing large timing tests but I did neither this time. As this can digress into a whole different off-question-topic soap opera I regret sharing my test results :( Is it too late to pretend this conversation never happened? FYI gmail message metadata size is 2.5 MB and 3 index files are about 800 KB. Also it's not a hard disk, it's a Samsung Pro 960 NVMe SSD on 4 channels. – WinEunuuchs2Unix Jun 26 at 23:43
  • 1
    It probably doesn't matter what storage device you have for these tests, if this is happening on a Linux machine. The kernel is extremely good at buffering files in memory. That's what the "buff/cache" value is when using the free command. Actual writes to the device take place at a moment chosen by an algorithm that takes into account age of the cache and memory pressure on the machine. If you're trying multiple tests, then the first file reads will come off the disk but subsequent reads will most likely come straight out of memory (see sync; echo 3 > /proc/sys/vm/drop_caches). – Mike S Jun 27 at 11:52
  • Yesterday I made few tests with large files over 2GB and - yes, this approach is relatively slow than using of the cp command, but I can't say there is any significant performance difference. – pa4080 Jun 28 at 14:08
12

You can use find:

find . -max-depth 1 -name 'gmail-meta3*' -exec cp "{}" "{}.bak" \;

That will find in the current directory . all files with a name matching the glob pattern (mind the single quotes around the pattern to prevent shell globbing). For each file found, it will exec cp from name to name.bak. The \; at the end ensures it will do each file individually instead of passing all of them at once. The max depth as 1 only searches the cuurent directory instead of recursing down.

  • 1
    Will it work with find . -max-depth 1 -name '"$1"'' -exec cp "{}" "{}$2" \; when $1 is source and $2 is extension? – WinEunuuchs2Unix Jun 26 at 3:27
  • $2 should be ok to substitute so long as it is reasonable. $1 could be trickier as we can't do variable substitution inside single quotes. I'm not sure offhand but it might be possible to use $1 in double quotes since the pattern is stored in a string. – cbojar Jun 26 at 3:36
11

You can use a for loop with bash. Normally, I would just type it as a one-liner because this isn't a task I perform often:

for f in test* ; do cp -a "$f" "prefix-${f}.ext" ; done

However, if you need it as a script:

cps() {
   [ $# -lt 2 ] && echo "Usage: cps REGEXP FILES..." && return 1

   PATTERN="$1" ; shift

   for file in "$@" ; do
      file_dirname=`dirname "$file"`
      file_name=`basename "$file"`
      file_newname=`echo "$file_name" | sed "$PATTERN"`

      if [[ -f "$file" ]] && [[ ! -e "${file_dirname}/${file_newname}" ]] ; then
         cp -a "$file" "${file_dirname}/${file_newname}"
      else
         echo "Error: $file -> ${file_dirname}/${file_newname}"
      fi
   done
}

Usage is similar to rename. To test:

pushd /tmp
mkdir tmp2
touch tmp2/test{001..100}     # create test files
ls tmp2
cps 's@^@prefix-@ ; s@$@.bak@' tmp2/test*    # create backups
cps 's@$@.bak@' tmp2/test*    # more backups ... will display errors
ls tmp2
\rm -r tmp2                   # cleanup
popd
  • FYI: $ time for f in gmail-meta3* ; do cp -a "$f" "${f}.bak" ; done = real 0m0.046s – WinEunuuchs2Unix Jun 26 at 23:18
  • Um no I don't want to optimize for time. It is 0.046 seconds which means 0 seconds for human perception. I was just trying to show how I was testing posted answers and passing along interesting tidbits to viewers who looked at the sed command above. Or at least I was interested in comparing sed to cp.... – WinEunuuchs2Unix Jun 26 at 23:25
  • Your solution with cp is faster than solution with sed though. So it's a cause for celebration :) – WinEunuuchs2Unix Jun 26 at 23:38
  • (1) -a is a non-standard test operator.  Why not use -e?  (2) “Unable to create temporary directory.” is a somewhat misleading error message.  (3) Why not just use mktemp -d?  (4) You should test exit statuses.  For example, you should say ! mkdir "$FOLDER" && echo "Unable to create temporary directory." && return 1 or mkdir "$FOLDER" || { echo "Unable to create temporary directory."; return 1;}.  Likewise for cp and rename (and maybe even pushd, if you want to be careful).  … (Cont’d) – G-Man Jun 27 at 15:43
  • (Cont’d) …  (5) Arrggghhhh!  Don’t say $@; say "$@". (5b) There’s no need to use { and } when you reference variables the way you are doing ("${FOLDER}", "${PATTERN}" and "${file}"); just do "$FOLDER", "$PATTERN" and "$file". (6) This assumes that files are in the current directory.  cps 's/$/.bak/' d/foo will copy d/foo to foo.bak in the current directory, not d/foo.bak. – G-Man Jun 27 at 15:43
6

The closest you will likely get to the DOS paradigm is mcp (from the mmv package):

mcp 'gmail-meta3*' 'gmail-meta3#1.bak'

If zsh is available, its contributed zmv module is perhaps a little closer:

autoload -U zmv

zmv -C '(gmail-meta3*)' '$1.bak'

I'd avoid ls regardless - a variant on your own answer that's safe for whitespace (including newlines) would be

printf '%s\0' gmail-meta3* | while IFS= read -r -d '' f; do cp -a -- "$f" "$f.bak"; done

or perhaps

printf '%s\0' gmail-meta3* | xargs -0 -I{} cp -a -- {} {}.bak
  • I understand mmv is the package but in comments you say the command is mcp but then in the command you use mmv which is also a command in mmv package. I like the direction of the printf examples and in a polished script I'd ensure $1 and $2 were passed. +1 for getting ball rolling :) – WinEunuuchs2Unix Jun 26 at 1:19
  • @WinEunuuchs2Unix apologies - the mcp/mmv was a brainfart. Actually mcp is just a synonym for mmv -c – steeldriver Jun 26 at 1:21
  • Pfft no worries. If I had a dollar for every typo I've made I'd be a millionaire :) I'd like clarification on printf command I've never used really. Are you saying printf '%s\0' "$1"* would work if gmail-meta3 was passed as parameter 1? – WinEunuuchs2Unix Jun 26 at 1:24
  • @WinEunuuchs2Unix I'd probably let the calling context do the globbing i.e. cps gmail-meta3* and then write printf '%s\0 "$@" | while ...` in the function. Or simply use for f; do cp -- "$f" "$f.bak"; done (like xiota's answer, but as a function) – steeldriver Jun 26 at 1:40
  • 1
    Note that with zmv you can use "wildcard replacement" mode, which I find a bit easier to grok: zmv -W -C 'gmail-meta3*' '*.bak' – 0x5453 Jun 27 at 22:03
5

rsync only solution

If you just want to backup your files, you can copy them to a new directory

rsync /path/to/dir/Filename* /path/to/backupdirectory

This will copy the Filename files from /path/to/dir/ to /path/to/backupdirectory.


rsync + filerename

If you want your backup files to have a suffix, things get hacky with rsync...

rsync -Iu /path/to/dir/Filename* /path/to/dir/Filename* -b --backup-dir=/path/to/backupdirectory --suffix=.bak

This would overwrite the existing files... with the existing files (-I) but only if they are (-u) newer (which they aren't) and creating a backup, with a suffix.

You can also do that in the same directory. But better exclude existing backups.

rsync -Iu /path/to/dir/Filename* /path/to/dir/Filename* -b --backup-dir=/path/to/backupdirectory --suffix=.bak --exclude '*.bak'

  • I love rsycnc so I upvoted but, a simpler method would be cp Filename* /path/to/backup/dir because files wouldn't need *.bak uniquifier if they were in a separate directory. – WinEunuuchs2Unix Jun 26 at 22:59
4

This one should do as requested:

cps(){ p="${@: -1}"; for f in "${@:1:$#-1}"; do cp -ai "$f" "${p//\?/$f}"; done  }

Usage:

cps FILES... pattern
Example 1: cps gmail-meta3* ?.bak
Example 2: cps * save-?
Example 3: cps * bla-?-blubb

I chose ? because # must be quoted when it's the first character of pattern, otherwise it's recognized as the start of a comment.

Test:

$ touch 'test};{bla#?"blubb'
$ cps test* bla-?-blubb
$ ls
test};{bla#?"blubb  bla-test};{bla#?"blubb-blubb


Some earlier versions of the script for adding a suffix:

Similar to @WinEunuuchs2Unix answer, but I think more flexible and not parsing ls:

cps(){ S="$1"; shift; printf '%s\0' "$@" | xargs -0 -I{} cp -abfS "$S" {} {}; }

Put this in your .bashrc.

Usage:

cps SUFFIX FILES...
Example: cps .bak gmail-meta3*

Alternative, with the suffix as last argument (via and via):

cps(){ S="${@: -1}"; printf '%s\0' "${@:1:$#-1}" | xargs -0 -I{} cp -abfS "$S" {} {}; }

Usage:

cps FILES... SUFFIX
Example: cps gmail-meta3* .bak

  • Nice coding but it's kind of hard after decades of using Source then Target to switch copy command to Target then Source – WinEunuuchs2Unix Jun 26 at 10:35
  • Added the function with the suffix in the back. – pLumo Jun 26 at 11:14
  • Thanks that is more intuitive. Calling it a suffix is accurate the way my answer coded it but it's really a target or destination. Other users might want to use: copy gmail-meta3* old-meta3*. In my answer I couldn't figure out how to get * into the destination name like my question requested... – WinEunuuchs2Unix Jun 26 at 11:22
  • problem is that * is interpreted by the shell, so the function will not know about it. You would need some other character or quote it, then replace it with the original filename within the function. – pLumo Jun 26 at 11:26
  • I guess the # could be used as a replacement wildcard for *? So you could type copy filenames# save-#. I think you'd want the wildcard character to be the same for source and target. – WinEunuuchs2Unix Jun 26 at 11:53
4

I wrote this one-liner into my ~/.bashrc. Much better answers using find can be posted I suppose. Even better answers could be written in in C. Hopefully this Q&A gets the ball rolling for better answers:

cps () {
    # cps "Copy Splat", copy group of files to backup, ie "cps Filename .bak"
    # Copies Filename1 to Filename1.bak, Filename2 to Filename2.bak, etc.
    # If Filename1.bak exists, don't copy it to Filename1.bak.bak
    for f in "$1"*; do [[ ! "$f" == *"$2" ]] && cp -a "$f" "$f$2"; done

    # OLD version comments suggested to remove 
    # ls "$1"* | while read varname; do cp -a "$varname" "$varname$2"; done
}
  • for f in "$1"*; do : $1 is the gmail-meta3 parameter and f is the list of files matching. Combined this means for gmail-meta3, gmail-meta3-LAB-9999, etc. do the following
  • [[ ! "$f" == *"$2" ]] && : $f is the same as f above. $2 is the .bak parameter passed. Combined this means if the filename doesn't end in .bak (because we don't want to copy .bak and create .bak.bak) then do the following
  • cp -a "$f" "$f$2"; copy gmail-meta3 to gmail-meta3.bak, etc.
  • done : loop back and grab next filename in gmail-meta3* list.

cps gmail-meta3 .bak Sample Output

Using the question as an example here is how it looks in action:

───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ ll gmail-meta3*
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3.bak
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821
-rw-rw-r-- 1 rick rick   728954 Jun 27 05:46 gmail-meta3-YAD-1558392194-26467821.bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ cps gmail-meta3 .bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ ll gmail-meta3*
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3.bak
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821.bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ 

Note: This uses the -a flag with the cp command to preserve timestamps and give you better grasp of your file backups.

Notice how the file copies have the exact same date and time as the originals. If the -a parameter was omitted they would be given the current date and time and wouldn't look like a true backup except that the file size would be the same.

  • 6
    don't people always recommend against parsing ls – qwr Jun 26 at 13:30
  • 3
    Since you mention find I assume you’re aware of the dangers of parsing ls? But in your case neither is necessary: just do for file in "$1"*; do copy -a "$file" "$file$2"; done — this is completely safe and much simpler than any kind of indirection via ls or find and a while loop. – Konrad Rudolph Jun 26 at 13:40
  • @KonradRudolph Thanks for your suggestion. I implemented and tested your suggestion with a couple minor changes. – WinEunuuchs2Unix Jun 27 at 23:21
2

Another method of achieving your requirement is to copy the files to a temporary directory and use the rename command to rename them.

$ mkdir backup
$ cp filename* /tmp/rename-backup/
$ rename 's/(filename.*)/$1.bak/' /tmp/rename-backup/*
$ mv /tmp/rename-backup/* ./

If you need it as a script, you can use it like so

cps () {
    mkdir -p /tmp/rename-backup/
    cp "$1"* /tmp/rename-backup/
    rename "s/($1.*)/\$1.$2/" /tmp/rename-backup/*
    mv "/tmp/rename-backup/$1"*".$2" .
}

And you can use it like so:

cps file bak

This is an example

$ ls -l
total 0
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file a
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ab
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ac
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename1
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename2
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename3
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename4
$ cps file bak
$ ls -l
total 0
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file a
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 file a.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ab
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 file ab.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ac
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 file ac.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename1
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 filename1.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename2
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 filename2.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename3
-rw-r--r--  1 danny  wheel  0 Jun 26 16:41 filename3.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename4
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 filename4.bak

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.