So here is the culprit:

zcat Merged.csv.gz | awk -F, 'FNR == 1  {header = $0; next} !seen[$7]++ {print header | "gzip > data/S20180528_"$7".txt.gz"} {print | "gzip > data/S20180528_"$7".txt.gz";}'

When I run it on this big file (Merged.csv.gz) I get:

awk: cannot open "gzip > data/S20180528_2505329.txt.gz" for output (Too many open files)

I created the original command by pasting parts I found here and on stackoverflow. Googling around, I think I might have messed up some of the quotes doing so. Now, I can't figure which ones.


That means you have to close the open file handles.

It might be cleaner to

zcat Merged.csv.gz | 
    awk -F, '
        FNR == 1  {header = $0; next} 
        !seen[$7]++ {
            file[$7] = "data/S20180528_"$7".txt"
            print header > file[$7]
        {print >> file[$7]; close(file[$7])}
        END {
            for (f in file) system("gzip " file[f])

Since the file is sorted on column 7:

zcat Merged.csv.gz | awk -F, '
    FNR == 1  {header = $0; next} 
    $7 != key {
        if (file) {
            system("gzip " file)
        file = "data/S20180528_" $7 ".txt"
        key = $7
        print header > file
    {print > file}
    END { close(file); system("gzip " file) }
  • this does not work for me. The code runs and prints errors message: gzip: 2504672: No such file or directory gzip: 2504665: No such file or directory gzip: 2504579: No such file or directory ... (the numbers there are the values of $7). There is a secondary problem here. The entries of $7 are sorted. So it should be possible to gzip the files as they are produced. It seems to me your solution produces all files in txt than zips them in one swoop. But the intermediate output (all those txt files) are really large. I would rather gzip them incrementally, as we go from one $7 to the next. – user2413 Oct 24 '18 at 0:20
  • 1
    I had an error in the last line, should be system("gzip " file[f]) not system("gzip " f). I'll think about your other requirements – glenn jackman Oct 24 '18 at 3:10
  • Thanks. Your code works great. My only regret is that I need to wait two more days before I can award it a bounty;) – user2413 Oct 24 '18 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.