5

In question How to rename all files in folder with name ending with "_backup" @Radu Rădeanu gave a good answer that was useful to me as well:

find . -type f -name '*.jpg_backup' -print0 \
| while IFS= read -r -d '' file ; do mv -- "$file" \
"$(echo $file | sed 's/_backup//g')"; done

However, I would like to fully understand his one-liner. Precisely, the part that I don't understand is:

while IFS= read -r -d '' file

I get that IFS is "internal field separator" and I suppose this is removing or ignoring the whitespace, but I don't understand the syntax and the options.

I would also like to understand why the -- is necessary after mv.

Could someone help? Thanks.

2 Answers 2

5

read word1 word2 … rest does the following:

  1. Read a line.
  2. While the current input ends in a backslash, read another line and append it to the first one, minus the backslash-newline.
  3. Break the input into separate words, where any character in the value of the IFS variable is considered a word separator.
  4. Assign the first word to the variable word, the second word to the variable word2, etc.
  5. If there are words left over, they are assigned to the variable rest, with the original word separators inside preserved.

The option -r deactivates step 2, so a \ at the end of a line is not considered a word separator.

With IFS set to the empty string, there are no word separators, so the whole line is one big word: step 3 does nothing, and step 5 ends up assigning the original line to the specified variable. Regarding why IFS is set in this location, see Why is while IFS= read used so often, instead of IFS=; while read..?.

The option -d changes the notion of line: normally a line ends in a newline character; with -d '' (empty argument to -d), bash reads input records delimited by a null byte instead.

The upshot is that find … -print0 | while IFS= read -r -d '' file; do … executes the loop body for each match that find prints, regardless of any special characters that may occur in the file name.


As for the -- in mv -- "$file", it's there in case the value of file begins with a dash: mv would interpret it as an option. In this particular case, it isn't necessary, because the ouptut of this find command always starts with ./. Using -- systematically in scripts is arguably good hygiene.


There is a defect in this snippet: it still won't cope with file names containing whitespace or \[?*, because of the "$(echo $file | sed 's/_backup//g')" bit. A variable expansion like $file doesn't just substitute the value of the variable: it splits the variable into words using the value of IFS (just like read does), and treats each word as a wildcard pattern that is replaced by the list of matching files if it matches any. To avoid this behavior, write "$file". This is a general shell programming rule: always put double quotes around variable substitutions (and command substitutions $(…) too), unless you know why you need to leave them out. (If you want the nitty-gritty, see When is double-quoting necessary?; $VAR vs ${VAR} and to quote or not to quote and What is the significance of single and double quotes in environment variables? may also be of interest).

The quick fix is to add the double quotes:

mv -- "$file" "$(echo "$file" | sed 's/_backup//g')"

This happens to work here because of the way the input is produced, but for general values of file it fails in a few rare cases:

  • If the value of file consists of the character - followed by one or more letters among Een, then echo will treat it as an option and output nothing.
  • Command substitution eats up the last newlines of the output, so if $file ends with one or more newline characters, they will be truncated.

Bash has a string substitution which can be used instead of sed here. It's more robust (you don't need to worry about subtleties with special characters) and faster.

mv -- "$file" "${file//_backup/}"

Though given what the question asked for, this is the wrong operation: it removes _backup anywhere in the file name instead of only at the end. Here doing it right would be easier.

mv -- "$file" "${file%_backup}"
2
  • +1 for responding to the important bugs in the original script, simplifying at the same time. Jan 7, 2014 at 15:01
  • Thank you very much for the explanation, Gilles. It's very clear now.
    – msoutopico
    Jan 12, 2014 at 22:39
1

In case you have a filename that happens to start with a hyphen, you need -- to separate mv options from the actual filename arguments.

If you have a filename that begins or ends with spaces, that incantation of read with IFS= will ensure the file variable contains the exact filename you want.

The -d '' option reads zero-byte separated "lines" instead of newline separated lines -- that's because you're using find's -print0 option.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .