I have CSVs from different directories that have the same filename and I would like to merge them together and place them into a new directory under the same filename, and then loop it through the source folders.

The CSVs has the same structure thru out from both sources.

I've been searching for a good solution for this for a while and my unix/bash-skills aren't as honed for this task as I want them to be.

  • 3
    Can you give a sample of two csv files? Do they have the same column delimiter and the same number of columns? Do they have a title row? – Byte Commander Sep 15 '16 at 10:09
  • 2
    Are you looking for something as simple as "find YOUR_BASEDIR -type f -exec cat {} >> YOUR_OUPUTFILE \;", e.g. just merging the contents of files with no regard for structure? – bgse Sep 15 '16 at 10:12
  • 2
    Please edit your question and show us an example of your input files and the output you would like to see. We need to know what exactly you mean by "merging". Do you just want one file after the other? How about headers? Or do you want each line to be appended (so all lines 1s become one huge line 1, the same for line 2 etc)? – terdon Sep 15 '16 at 10:22
  • The CSVs are identical in their structure. – F.Dahlberg Sep 15 '16 at 10:57

Normally a CSV file looks somehow like this:



Now if we have a second CSV file with compatible format and dimensions (same column delimiter, here , - and same number of columns, here 5) and without a title row containing headlines for each column, like the example below:



Then we can simply concatenate the two files without need for any conversion:

cat file1.csv file2.csv > output.csv

The result would be this:



To automatically merge all files in source1/ with equally named files in source2/ (assuming there are only CSV files and that all files in source1/ must also exist in source2/), storing the result in output/ with the same file name, the following Bash one-liner will do the job:

for f in source1/* ; do fname="$(basename -- "$f")" ; cat -- "source1/$fname" "source2/$fname" > "output/$fname" ; done
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.