1

Since my nfs is a win type of server, it can't contain :. I want to copy multiple files from source server to the nfs directory on another server and rename them at the same time.

The script looks like:

#!/bin/bash

in_files="node1:~/experiment/wrfout_d01_2012-12-01*"
###########################################################
# the files look like wrfout_d01_2012-12-01_00:00:00
# wrfout_d01_2012-12-01_06:00:00
# wrfout_d01_2012-12-01_12:00:00
# wrfout_d01_2012-12-01_18:00:00
###################################
for fn in $in_files;do
  echo $fn
  newfn=$(basename "$fn")
  echo 'apply basename :'
  echo $newfn
  scp -r "$fn"  "./${newfn//:/_}"
  echo 'next:'
done

After run this script, it reports

node1:~/experiment/wrfout_d01_2012-12-01*
apply basename :
wrfout_d01_2012-12-01_12_00_00
./wrfout_d01_2012-12-01*: File name too long
./wrfout_d01_2012-12-01*: File name too long
./wrfout_d01_2012-12-01*: File name too long
./wrfout_d01_2012-12-01*: File name too long
next:

But it seems that if I apply the script to only one file, it is OK,(i.e. change the in_files to a certain file),for example:

node1:~/experiment/wrfout_d01_2012-12-01_00:00:00
apply basename :
wrfout_d01_2012-12-01_00:00:00
wrfout_d01_2012-12-01_00:00:00        100%   71MB  70.5MB/s   00:00    
next:

1.So how should I do to let it work for multiple files?

2.By the way it seems rsync can also do this part, which one is faster or preferred.

1
  • Rsync is always preferred.
    – terdon
    Jun 4, 2019 at 9:37

1 Answer 1

0

Your target files are on another system. Therefore, your glob (node1:~/experiment/wrfout_d01_2012-12-01*) is never expanded. You can see this right there in the output you posted. The first line of output comes from echo $fn and it prints this:

node1:~/experiment/wrfout_d01_2012-12-01*

You are not iterating over ever file matching that glob, you are only processing the glob itself as a string. So, instead of attempting to list files that you don't yet have access to (you need to open the ssh connection to see the list of files), just bring everything over and rename locally:

#!/bin/bash

rsync -v node1:"~/experiment/wrfout_d01_2012-12-01*" .
rename 's/:/_/g' wrfout_d01_2012-12-01*

Alternatively, assuming your file names don't contain newline characters, you can do this:

#!/bin/bash

in_files=($(ssh node1 'ls ~/experiment/wrfout_d01_2012-12-01*') )
for fn in "${in_files[@]}";do
  newfn=$(basename "$fn")
  rsync node1:"$fn"  "${newfn//:/_}"
done
2
  • But, in this nfs directory, it would report Input/output error if the filename contain :. That is why I want to rename it at the same time of copy. It is noted that if the in_files is only a filename, it works. So the problem may become that can we obtain a list of filenames from source server? Jun 4, 2019 at 10:55
  • @AllenZhang then you need to ssh to node1, get the files and then copy. See updated answer.
    – terdon
    Jun 4, 2019 at 15:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .