8

I have a text file with 2 million lines. Each line has a positive integer. I am trying to form a frequency table kind of thing.

Input file:

3
4
5
8

Output should be:

3
7
12
20

How do I go about doing this?

  • 1
    In your text, you say, that you want a frequency table. Your output sample is a list. Can you please clarify this? – Wayne_Yux Feb 2 '17 at 8:34
  • Indeed your output is not a frequency table – don.joey Feb 2 '17 at 8:41
  • I'm sorry. I meant a cumulative frequency table. Have modified the question. Thanks. – user110327 Feb 2 '17 at 10:18
  • It's not very cool but I usually just do stuff like this into a spreadsheet. – John U Feb 3 '17 at 15:05
  • @JohnU I usually do, but the file I have has 1 million numbers. – user110327 Feb 6 '17 at 9:34

10 Answers 10

17

With awk:

awk '{total += $0; $0 = total}1'

$0 is the current line. So, for each line, I add it to the total, set the line to the new total, and then the trailing 1 is an awk shortcut - it prints the current line for every true condition, and 1 as a condition evaluates to true.

  • Please could you explain your code? – George Udosen Feb 2 '17 at 8:43
  • Can the word print be used also? – George Udosen Feb 2 '17 at 8:45
  • Yes, print total}, instead of $0 = total}1 – muru Feb 2 '17 at 8:47
  • 1
    @George ah, no. – muru Feb 2 '17 at 8:53
  • 6
    A shorter and perhaps more understandable way of writing the awk script would be {print(total += $0)} – Miles Feb 4 '17 at 3:25
9

In a python script:

#!/usr/bin/env python3
import sys

f = sys.argv[1]; out = sys.argv[2]

n = 0

with open(out, "wt") as wr:
    with open(f) as read:
        for l in read:
            n = n + int(l); wr.write(str(n)+"\n")

To use

  • Copy the script into an empty file, save it as add_last.py
  • Run it with the source file and targeted output file as arguments:

    python3 /path/to/add_last.py <input_file> <output_file>
    

Explanation

The code is rather readable, but in detail:

  • Open output file for writing results

    with open(out, "wt") as wr:
    
  • Open input file for reading per line

    with open(f) as read:
        for l in read:
    
  • Read the lines, adding the value of the new line to the total:

    n = n + int(l)
    
  • Write the result to the output file:

    wr.write(str(n)+"\n")
    
  • 2
    The code can be simplified – jfs Feb 2 '17 at 20:33
  • 3
    It is not about shortness or time performance (million lines is not big data). The code in your answer is not idiomatic Python. My answer is just more pythonic version of yours. – jfs Feb 2 '17 at 21:16
  • 8
    @J.F.Sebastian if the more idiomatic version is slower why would anyone prefer it? There's nothing special about being "pythonic" that's just a convention that helps python devs share code and standards for readability. If the more idiomatic version is less efficient (slower) then it shouldn't be used unless you're working in an environment where standardization is more important than performance (which sounds like a horrible idea to me). – terdon Feb 3 '17 at 8:55
  • 2
    @terdon there is something to be said about premature optimisation. Readability can be important because of long-term maintainability. – muru Feb 3 '17 at 9:04
  • 4
    @muru sure, but this is perfectly readable. It's only crime is not being "pythonic". Not to mention that we're talking about 7 lines of code, not some giant project. Sacrificing efficiency in the name of style conventions seems like the wrong approach. – terdon Feb 3 '17 at 9:10
9

Just for fun

$ sed 'a+p' file | dc -e0 -
3
7
12
20

This works by appending +p to each line of the input, and then passing the result to the dc calculator where

   +      Pops two values off the stack, adds them, and pushes the result.
          The precision of the result is determined only by the values  of
          the arguments, and is enough to be exact.

then

   p      Prints  the  value on the top of the stack, without altering the
          stack.  A newline is printed after the value.

The -e0 argument pushes 0 onto the dc stack to initialize the sum.

  • Something like this might actually be the fastest over a large dataset – Digital Trauma Feb 3 '17 at 2:26
  • @DigitalTrauma on a 1.3 million lines, actually almost the slowest: real 0m4.234s – Jacob Vlijm Feb 3 '17 at 6:14
  • fun is all it needs for an upvote :D quirky is enough too :D :D – Rinzwind Feb 3 '17 at 8:37
  • Please explain it a little. – AmanicA Feb 7 '17 at 3:25
8

In Bash:

#! /bin/bash

file="YOUR_FILE.txt"

TOTAL=0
while IFS= read -r line
do
    TOTAL=$(( TOTAL + line ))
    echo $TOTAL
done <"$file"
  • bash is extremely slow on this: real 0m53.116s, almost a minute, on 1.3 million lines :) – Jacob Vlijm Feb 3 '17 at 6:22
  • @JacobVlijm dash is about twice as fast, busybox ash and zsh (in sh mode) 1.5 times, but of course, even dash is 5 times slower than python. – muru Feb 3 '17 at 9:04
6

To print partial sums of integers given on the standard input one per line:

#!/usr/bin/env python3
import sys

partial_sum = 0
for n in map(int, sys.stdin):
    partial_sum += n
    print(partial_sum)

Runnable example.

If for some reason the command is too slow; you could use the C program:

#include <stdint.h>
#include <ctype.h>
#include <stdio.h>

int main(void)
{
  uintmax_t cumsum = 0, n = 0;
  for (int c = EOF; (c = getchar()) != EOF; ) {
    if (isdigit(c))
      n = n * 10 + (c - '0');
    else if (n) { // complete number
      cumsum += n;
      printf("%ju\n", cumsum);
      n = 0;
    }
  }
  if (n)
    printf("%ju\n", cumsum + n);
  return feof(stdin) ? 0 : 1;
}

To build it and run, type:

$ cc cumsum.c -o cumsum
$ ./cumsum < input > output

Runnable example.

UINTMAX_MAX is 18446744073709551615.

The C code is several times faster than the awk command on my machine for the input file generated by:

#!/usr/bin/env python3
import numpy.random
print(*numpy.random.random_integers(100, size=2000000), sep='\n')
5

You probably want something like this:

sort -n <filename> | uniq -c | awk 'BEGIN{print "Number\tFrequency"}{print $2"\t"$1}'

Explanation of the command:

  • sort -n <filename> | uniq -c sorts the input and returns a frequency table
  • | awk 'BEGIN{print "Number\tFrequency"}{print $2"\t"$1}' turns the ooutput into a nicer Format

Example:
Input File list.txt:

4
5
3
4
4
2
3
4
5

The command:

$ sort -n list.txt | uniq -c | awk 'BEGIN{print "Number\tFrequency"}{print $2"\t"$1}'
Number  Frequency
2   1
3   2
4   4
5   2
  • I like this the out put is nice :)... – George Udosen Feb 2 '17 at 8:57
5

You can do this in vim. Open the file and type the following keystrokes:

qaqqayiwj@"<C-a>@aq@a:wq<cr>

Note that <C-a> is actually ctrl-a, and <cr> is carriage return, i.e. the enter button.

Here's how this works. First off, we want to clear out register 'a' so that it has no side-effects on the first time through. This is simply qaq. Then we do the following:

qa                  " Start recording keystrokes into register 'a'
  yiw               " Yank this current number
     j              " Move down one line. This will break the loop on the last line
      @"            " Run the number we yanked as if it was typed, and then
        <C-a>       " increment the number under the cursor *n* times
             @a     " Call macro 'a'. While recording this will do nothing
               q    " Stop recording
                @a  " Call macro 'a', which will call itself creating a loop

After this recursive macro is done running, we simply call :wq<cr> to save and quit.

  • 1
    +1 for breaking down the magic incantation and explaining all the parts. Far too rare round these parts. – John U Feb 3 '17 at 15:00
5

Perl one-liner:

$ perl -lne 'print $sum+=$_' input.txt                                                                
3
7
12
20

With 2.5 million lines of numbers, it takes about 6.6 seconds to process:

$ time perl -lne 'print $sum+=$_' large_input.txt > output.txt                                        
    0m06.64s real     0m05.42s user     0m00.09s system

$ wc -l large_input.txt
2500000 large_input.txt
  • real 0m0.908s, quite nice. – Jacob Vlijm Feb 2 '17 at 21:36
  • @JacobVlijm that's on a pretty small file. I added a small test with 2.5 million lines file. 6.64 seconds – Sergiy Kolodyazhnyy Feb 2 '17 at 21:46
  • 1
    I ran 1.3 million lines on an ancient system – Jacob Vlijm Feb 2 '17 at 21:47
3

A simple Bash one-liner:

x=0 ; while read n ; do x=$((x+n)) ; echo $x ; done < INPUT_FILE

x is the cumulated sum of all numbers from the current line and above.
n is the number in the current line.

We loop over all the lines n of INPUT_FILE and add their numeric value to our variable x and print that sum during each iteration.

Bash is a bit slow here though, you can expect this to run around 20-30 seconds for a file with 2 million entries, without printing the output to the console (which is even slower, independend of the method you use).

3

Similar to @steeldriver's answer, but with the slightly less arcane bc instead:

sed 's/.*/a+=&;a/' input | bc

The nice thing about bc (and dc) is that they are arbitrary precision calculators, so will never overflow or suffer lack of precision over integers.

The sed expression transforms the input to:

a+=3;a
a+=4;a
a+=5;a
a+=8;a

This is then evaluated by bc. The a bc variable is auto-initialised to 0. Each line increments a, then explicitly prints it.

  • real 0m5.642s on 1.3 million lines. sed is really slow on this. – Jacob Vlijm Feb 3 '17 at 6:17

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.