Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am beginner in working with bash script. I have a data file:

i    z    r                    
0   -30   3.5                 
0   -29.5 3.4                 
0   -29   3.6                  
0    29.5 4.2                  
0    30   4.6                 
1   -30   2.8                  
1   -29.5 3.4                    
1    30   5.2
2   -30   2.5
2   -29.5 2.6

And I'm trying to rearrange data into a file like this:

i     z     r
0    -30    3.5
1    -30    2.8
2    -30    2.5
0    -29.5  3.4
1    -29.5  3.4
2    -29.5  2.6

I used the script below but it cant not run like what i desire.

awk '{
  for(z=/-30/; z<=30; z+=0.5){
    for(i=0; i<=17489;i++){
      if($1==$i  && $2==$z)
        print $1 $2 $3
}' data.dat >> avg_SD.dat

please help me! Thank you so much.

share|improve this question
I've edited as best I can to make this make sense but you need to explain the transformation between the two files. What is it that you're trying to do? – Oli Jun 23 '14 at 9:55
Thank for your edition. It looks better. In the above form: in per variale $i (i from 0 to 17489, step 1), i have calculate z and r value, respectively (z from -30 to 30, step 0.5). after finish z loop, the i was increased and repeat z loop. know, I want to modified in this way: in per variable $z (z from -30 to 30, step 0.5), there will have a r value per i frame. That is what i trying to do. – user297072 Jun 23 '14 at 10:15
How did you calculate the r value? And why sometimes "z += 0.5" and sometimes not and why the sign of z somtimes is changed? Your example is not really clear i can't see any logic how you calculate the new z and the new r values. – TuKsn Jun 23 '14 at 10:35
I use another script to calculate r and file out of this script is data.dat (above form). However calculating the average and standard divition of r value are difficult. So I have to modify it into below form. – user297072 Jun 23 '14 at 10:37
Ok if I understand you right, you are trying to loop through each line of your data.dat file, increase i+1 (in each block) and send z and r to another script (or function) to calculate them, then write the result of the changes in a new file? – TuKsn Jun 23 '14 at 10:48

From the example and your script it seems, that your original file is sorted by the first column ( i ) but you want it to be primarily sorted by the second column ( z ) instead and than by i.

This can be achieved much easier with sort

tail -n +2 data.dat | sort -n -k 2 -k 1 >> avg_SD.dat

It is also probably much faster than looping through two loops and doing at least one million comparisions for every line in the input file.


  • tail -n +2 data.dat prints data.dat starting with the second line, thus removing the header.

  • | redirects the output of the command on the left to the input of the command on the right.

  • sort -n -k 2 -k 1 sorts the input numerically (-n), first by the second field (-k 2) than by the first field (-k 1)

  • >> avg_SD.dat prints the output to avg_SD.dat keeping previous content. If you want to overwrite the previous content, use a single > instead of >>.

If there is no header in the original file, you do not need tail and can pass data.dat directly to sort as an argument:

sort -n -k 2 -k 1 data.dat >> avg_SD.dat
share|improve this answer
thank Adaephon so much. It works. If I want my file start form z = -30 and finish when z = 30, how can I do? other hand, can I use sort command to calculate average and Standard division for r. for ex: i want to calculate r for every frame number 0. – user297072 Jun 23 '14 at 11:59
To select only lines with -30 <= z <= 30 you can use awk '(-30 <= $2 && $2 <= 30) {print $0}'. And no, sort is only good at sorting. awk could be used to do these calculations, but it seems to me that a more general programming language might serve your needs better, for example python or perl. – Adaephon Jun 25 '14 at 20:12

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.