15

I have a list of filenames inside a file called list_of_files.txt.

I want to copy the contents of each file in that list into another file called all_compounds.sdf.

How should I do this from the command line?

20

Don't use simple command substitution to get filenames (that could easily break with spaces and other special characters). Use something like xargs:

xargs -d '\n' -a list_of_files.txt cat > all_compounds.sdf

Or a while read loop:

while IFS= read -r file; do cat "$file"; done < list_of_files.txt > all_compounds.sdf

To use command substitution safely, at least set IFS to just the newline and disable globbing (wildcard expansion):

(set -f; IFS=$'\n'; cat $(cat list_of_files.txt) > all_compounds.sdf)

The surrounding parentheses () are to run this in a subshell, so that your current shell isn't affected by these changes.

14

Quick and dirty way...

cat $(cat list_of_files.txt) >> all_compounds.sdf

Please note: this only works if the filenames in your list are very well behaved - things will go wrong if they have spaces, newlines, or any characters that have special meaning to the shell - use this answer instead for reliable results)

Notes

  • cat concatenates files. It also prints their contents.

  • Using command substitution command2 $(command1) you can pass the output of command1 (cat list...) to command2 (cat) which concatenates the files.

  • Then use redirection >> to send the output to a file instead of printing to stdout. If you want to see the output, use tee instead:

      cat $(cat list_of_files.txt) | tee -a all_compounds.sdf
    

(I have used >> instead of > and tee with the -a switch in case your file already exists - this appends to the file instead of overwriting it, if it already exists)

4
  • 1
    @Zanna quote the command substitutions to avoid word splitting, like "$(cat list_of_files.txt)" Jan 3 '17 at 7:51
  • 4
    @Serg if word splitting is not done, then cat gets the entire list as one argument.
    – muru
    Jan 3 '17 at 7:54
  • @muru OK, how do we deal with filenames that contain spaces, then ? Jan 3 '17 at 8:10
  • 1
    @Serg set IFS accordingly - see the last paragraph of my answer
    – muru
    Jan 3 '17 at 8:15
4

While GNU awk is a text processing utility, it allows running external shell commands via system() call. We can utilize that to our advantage like so:

$ awk '{cmd=sprintf("cat \"%s\"",$0); system(cmd)}' file_list.txt                                                        

The idea here is simple: we read the file line by line, and out of each line we create formatted string cat "File name.txt", which is then passed to system().

And here it is in action:

$ ls
file1.txt  file2.txt  file3 with space.txt  file_list.txt


$ awk '{cmd=sprintf("cat \"%s\"",$0); system(cmd)}' file_list.txt                                                        
Hi, I'm file2
Hi, I'm file1
Hi, I'm file3

So we've done the big part of the task there already - we printed all the files on the list. The rest is simple : redirect final output to file with > operator into the summary file.

awk '{cmd=sprintf("cat \"%s\"",$0); system(cmd)}' file_list.txt > output.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.