2

As we process our csv data, we generate a lot of output files with 30 000 lines in each one of them. They all have the same columns/fields. They are all also in csv format and we put them into the same folder on the Linux server. The files are uniquely named using a combination of date, time and numeric digits. See below.

AB_20151127_120000_0_SEGMENT_FINAL.csv
AB_20151127_120000_1_SEGMENT_FINAL.csv
AB_20151127_120000_2_SEGMENT_FINAL.csv
AB_20151127_120000_3_SEGMENT_FINAL.csv
.
.
.
AB_20151127_120000_599_SEGMENT_FINAL.csv

So now we need to merge/join all of them into one big file called: AB_20151127_120000_SEGMENT_FINAL.csv (note the missing numeric digits from the merged file)

I tried awk as below but it is not working. Please tell me what I did wrong.

awk '"AB_20151127_120000_" NR-1 "_SEGMENT_FINAL.csv"' > AB_20151127_120000_SEGMENT_FINAL.csv
  • Are you looking for cat AB_20151127_120000_{1..599}_SEGMENT_FINAL.csv > AB_20151127_120000_SEGMENT_FINAL.csv? – muru Dec 9 '15 at 10:48
  • Thank you @muru. Let me try that quickly. Do I have to specify {1..599} or can I do something like {*} to say however many files there might be? – Gert Myburgh Dec 9 '15 at 10:55
  • If you use AB_20151127_120000_*_SEGMENT_FINAL.csv, the files will be in lexicographic order (1, 11, 12, ..., 199, 2, 21, ...). – muru Dec 9 '15 at 10:57
  • O I see. Your suggestion to use 'cat' is working. Thank you @muru. I just now need to know how many files gets generated when I originally split the files using 'awk'. This is all going to happen on the fly and I will have to work something into my script to see how many files were created. I can't just look at the file size and divide by 30 000 because although all of them have the same columns, the data size in those columns might differ from one file to the other. – Gert Myburgh Dec 9 '15 at 11:08
  • replace 599 with $(ls AB_20151127_120000_*.csv | wc -l) – Ralph Rönnquist Dec 9 '15 at 11:14
3

If the order in which the files are concatenated is not important, use:

cat AB_20151127_120000_*_SEGMENT_FINAL.csv > AB_20151127_120000_SEGMENT_FINAL.csv

If the order is important, you'll have to get creative. If you know the number of segments, 599 for example, you can use brace expansion (the \ is only there to let me print the command on two lines for readability):

cat AB_20151127_120000_{0..599}_SEGMENT_FINAL.csv > \
    AB_20151127_120000_SEGMENT_FINAL.csv

If you don't, you can still use brace expansion. Just choose a large enough number to be sure that all files will be included and ignore error messages about non-existant files:

cat AB_20151127_120000_{0..599}_SEGMENT_FINAL.csv > \
    AB_20151127_120000_SEGMENT_FINAL.csv 2>/dev/null

Alternatively, you can generate a list of sorted file names and use that:

cat $(printf '%s\n' AB_20151127_120000_*_SEGMENT_FINAL.csv | sort -nt_ -k4) > \
    AB_20151127_120000_SEGMENT_FINAL.csv

The printf will print each file name followed by a newline which is the passed to sort which will sort it numerically (-n) on the 4th field (-t4) where fields are defined by _ (-t_).

  • Thank you @terdon. Much appreciated. Works like a charm. – Gert Myburgh Dec 9 '15 at 12:43
0

If you have access to a Zsh shell, the task can be reduced to a single command:

cat AB_20151127_120000_*(n)_SEGMENT_FINAL.csv >AB_20151127_120000_SEGMENT_FINAL.csv

This is because the (n) globbing qualifier forces the * globbing pattern to expand to a list of filenames sorted in their natural order, as opposed to their lexicographical order.

For comparison, filename expansion in Bash:

$ for f in *; do echo "$f"; done
AB_20151127_120000_0_SEGMENT_FINAL.csv
AB_20151127_120000_10_SEGMENT_FINAL.csv
AB_20151127_120000_1_SEGMENT_FINAL.csv
AB_20151127_120000_2_SEGMENT_FINAL.csv
AB_20151127_120000_3_SEGMENT_FINAL.csv

Filename expansion in Zsh using the (n) globbing qualifier:

% for f in *(n); do echo "$f"; done
AB_20151127_120000_0_SEGMENT_FINAL.csv
AB_20151127_120000_1_SEGMENT_FINAL.csv
AB_20151127_120000_2_SEGMENT_FINAL.csv
AB_20151127_120000_3_SEGMENT_FINAL.csv
AB_20151127_120000_10_SEGMENT_FINAL.csv

protected by Community Dec 10 '15 at 6:23

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.