7

In bash I can grep some time measurements from a log file like this

grep "time:" myLogfile.txt | cut -d' ' -f 3 >> timeMeasurements.txt

#timeMeasurements.txt
2.5
3.5
2.0
...

Now I would like to compute the mean value from the values in timeMeasurements.txt. What is the quickest way to do that in bash?
I know that there is gnuplot and R but it seems like one has to write some lengthy script for either one on them.

11

Another way, using sed and bc:

sed 's/^/n+=1;x+=/;$ascale=1;x/n' timemeasurements.txt | bc

The sed expression converts the input to something like this:

n+=1;x+=2.5
n+=1;x+=3.5
n+=1;x+=2.0
scale=1;x/n

This is piped to bc which evaluates it line-by-line.

13

Obligatory GNU datamash version

$ datamash mean 1 < file
2.6666666666667

ASIDE: it feels like this really should be possible natively in bc (i.e. without using the shell, or an external program, to loop over input values). The GNU bc implementation includes a read() function - however it appears to be frustratingly difficult to get it to detect end-of-input. The best I could come up with is:

#!/usr/bin/bc

scale = 6
while( (x = read()) ) {
  s += x
  c += 1
}
s/c
quit

which you can then pipe file input to provided you terminate input with any non-numeric character e.g.

$ { cat file; echo '@'; } | ./mean.bc
2.666666
  • 1
    Not only shortest answer but exposes those interested in stats to a handy utility +1 – WinEunuuchs2Unix Apr 5 '17 at 12:44
  • This looks intriguingly short. Is there an apt-get version of datamash? – mcExchange Apr 5 '17 at 19:57
  • @mcExchange sudo apt install datamash was sufficient on my Ubuntu 16.04 VM. – Digital Trauma Apr 5 '17 at 21:08
  • Unfortunately on Ubuntu 14.04 it's not in the repositories ... – mcExchange Apr 5 '17 at 22:20
12

You could use awk. Bash itself is not very good at maths...

awk 'BEGIN { lines=0; total=0 } { lines++; total+=$1 } END { print total/lines }' timeMeasurements.txt

Notes

  • lines=0; total=0 set variables to 0
  • lines++ increase lines by one for each line
  • total+=$1 add the value in each line to the running total
  • print total/lines when done, divide the total by the number of values
  • 3
    Although it's good practice, variables don't actually need to be initialized in awk - so you could "golf" this to awk '{total+=$1} END{print total/NR}' – steeldriver Apr 5 '17 at 12:33
  • @steeldriver thanks! Just starting to learn :) – Zanna Apr 5 '17 at 12:38
8

Adapting the R command from this U&L post:

$ Rscript -e 'd<-scan("stdin", quiet=TRUE)' -e 'cat(mean(d), sep="\n")' < foo
2.666667
5

You can use bc the basic calculator, in a while loop with read:

count=0; sum=0; while read -r num; do ((count++)); sum=$(echo "$sum + $num" | bc); done < timeMeasurement.txt; echo "scale=2; $sum / $count" | bc -l

Or more readably:

count=0
sum=0
while read -r num
do
  ((count++))
  sum=$(echo "$sum + $num" | bc)
done < timeMeasurement.txt
echo "scale=2; $sum / $count" | bc -l

Explanation:

  • First we set the count of values and the sum total as the variables count and sum, with values of 0.
  • Read the file line by line, setting the value in the line as the variable num. We use the construction while read -r num; do ... ; done < timeMeasurements.txt to do this. This will mean that we'll do something for each line of the file.
  • Within the while loop, increment the count variable by one for each line with bash arithmetic ((count++)).
  • Use bash command substitution $(...) with echo piped to bc to add the value of the num variable for this line of the file, to the sum of the num variable from all previous lines. bc is used as bash does not cope well with floating point arithmetic.

At this point the loop ends, the count variable contains the number of time measurement values, the sum variable contains the sum of the time measurements.

  • Use echo with our variables to create the mean calculation which is passed to bc. The scale=2 part tells bc how many significant figures to display.
4

The datamash one seems a good option, but even acknowledging that my answer may be overkill, just in case you want to do a bit more than just a mean, octave is not so verbose:

$ octave
octave:1> load timeMeasurements.txt 
octave:2> mean(timeMeasurements)
ans =  2.6667

If you are doing means, remember that the same mean could come from very different behaviours, so the standard deviation is usually also relevant:

octave:3> std(timeMeasurements)
ans =  0.76376

or even a simple histogram is easy to do:

octave:4> hist(timeMeasurements)

Also, I think datamash is not in the apt-get repositories for trusty, only for newer versions.

Edit:

Oneliner, for more script-friendly usages:

octave -q --eval "m = load(\"timeMeasurements.txt\"); mean(m)"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.