I have a large data file and want to split it into smaller files based on values in column 1. For example column 1 has numbers from 1 to 10 ten times to make 100 rows and I want all lines with numbers '1' or '2' or '3' etc in their own file (preferably without sorting). Also I dont want to run the command 10 times so would like for it to be in a loop.

My files look like this:

  • text.txt

    enter image description here

  • ID.txt


The command I tried:

cat ID.txt | while read line; do awk '$1 == ${line}' test.txt >$line.txt;done

So to summarize I want it to read the value from the ID.txt file e.g '1' and then extract all rows with '1' in the first line and put it in a file called 1.txt, then it iterates to 2 then 3 then 4 etc.

But somehow the '$1 == ${line}' part I think is not working

  • 2
    Please include an example of your actual file. Not a screenshot of it, the text itself. You can use the formatting tools to make it look like code. – terdon Oct 21 '15 at 13:28

You are looking for the -v option to awk:

   -v var=val
   --assign var=val
          Assign the value val to the variable var,  before  execution  of
          the  program  begins.  Such variable values are available to the
          BEGIN rule of an AWK program.

Something like this:

cat ID.txt | 
    while read line; do awk -vline="$line" '$1 == l' test.txt >"$line".txt;done

Which would be better expressed as (avoiding the useless use of cat):

while read line; do 
    awk -vline="$line" '$1 == l' test.txt >"$line".txt;
done < ID.txt

However, this is very slow and inefficient. You are running the awk command on the entire test.txt for each line of ID.txt. Why not just read ID.txt in awk itself and print the matching lines:

awk 'NR==FNR{a[$1]++; next} ($1 in a){print >> $1".txt"}' ID.txt test.txt 

The above saves the 1st field of ID.txt in the array a. NR and FNR are special awk variables meaning "the current line of the input stream" and "the current line of the current file". The two will only be equal to each other when the first file is being read. Therefore, NR==FNR{a[$1]++; next} will only be run on the lines of the first file. The second part will not be executed because the next tells awk to skip to the next line.

The second part, checks if the current line's 1st field (remember, this is only run on the second file) exists in the array a (which means it was in ID.txt) and, if it does, prints the line into a file called "field1.txt"


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.