# How do I generate a running cumulative total of the numbers in a text file?

I have a text file with 2 million lines. Each line has a positive integer. I am trying to form a frequency table kind of thing.

Input file:

``````3
4
5
8
``````

Output should be:

``````3
7
12
20
``````

How do I go about doing this?

• In your text, you say, that you want a frequency table. Your output sample is a list. Can you please clarify this? Feb 2, 2017 at 8:34
• Indeed your output is not a frequency table Feb 2, 2017 at 8:41
• I'm sorry. I meant a cumulative frequency table. Have modified the question. Thanks.
– user590995
Feb 2, 2017 at 10:18
• It's not very cool but I usually just do stuff like this into a spreadsheet. Feb 3, 2017 at 15:05
• @JohnU I usually do, but the file I have has 1 million numbers.
– user590995
Feb 6, 2017 at 9:34

With `awk`:

``````awk '{total += \$0; \$0 = total}1'
``````

`\$0` is the current line. So, for each line, I add it to the `total`, set the line to the new `total`, and then the trailing `1` is an awk shortcut - it prints the current line for every true condition, and `1` as a condition evaluates to true.

• Please could you explain your code? Feb 2, 2017 at 8:43
• Can the word `print` be used also? Feb 2, 2017 at 8:45
• Yes, `print total}`, instead of `\$0 = total}1`
– muru
Feb 2, 2017 at 8:47
• @George ah, no.
– muru
Feb 2, 2017 at 8:53
• A shorter and perhaps more understandable way of writing the awk script would be `{print(total += \$0)}` Feb 4, 2017 at 3:25

In a python script:

``````#!/usr/bin/env python3
import sys

f = sys.argv[1]; out = sys.argv[2]

n = 0

with open(out, "wt") as wr:
n = n + int(l); wr.write(str(n)+"\n")
``````

### To use

• Copy the script into an empty file, save it as `add_last.py`
• Run it with the source file and targeted output file as arguments:

``````python3 /path/to/add_last.py <input_file> <output_file>
``````

### Explanation

The code is rather readable, but in detail:

• Open output file for writing results

``````with open(out, "wt") as wr:
``````
• Open input file for reading per line

``````with open(f) as read:
``````
• Read the lines, adding the value of the new line to the total:

``````n = n + int(l)
``````
• Write the result to the output file:

``````wr.write(str(n)+"\n")
``````
• The code can be simplified
– jfs
Feb 2, 2017 at 20:33
• It is not about shortness or time performance (million lines is not big data). The code in your answer is not idiomatic Python. My answer is just more pythonic version of yours.
– jfs
Feb 2, 2017 at 21:16
• @J.F.Sebastian if the more idiomatic version is slower why would anyone prefer it? There's nothing special about being "pythonic" that's just a convention that helps python devs share code and standards for readability. If the more idiomatic version is less efficient (slower) then it shouldn't be used unless you're working in an environment where standardization is more important than performance (which sounds like a horrible idea to me). Feb 3, 2017 at 8:55
• @terdon there is something to be said about premature optimisation. Readability can be important because of long-term maintainability.
– muru
Feb 3, 2017 at 9:04
• @muru sure, but this is perfectly readable. It's only crime is not being "pythonic". Not to mention that we're talking about 7 lines of code, not some giant project. Sacrificing efficiency in the name of style conventions seems like the wrong approach. Feb 3, 2017 at 9:10

Just for fun

``````\$ sed 'a+p' file | dc -e0 -
3
7
12
20
``````

This works by appending `+p` to each line of the input, and then passing the result to the `dc` calculator where

``````   +      Pops two values off the stack, adds them, and pushes the result.
The precision of the result is determined only by the values  of
the arguments, and is enough to be exact.
``````

then

``````   p      Prints  the  value on the top of the stack, without altering the
stack.  A newline is printed after the value.
``````

The `-e0` argument pushes `0` onto the `dc` stack to initialize the sum.

• Something like this might actually be the fastest over a large dataset Feb 3, 2017 at 2:26
• @DigitalTrauma on a 1.3 million lines, actually almost the slowest: `real 0m4.234s` Feb 3, 2017 at 6:14
• fun is all it needs for an upvote :D quirky is enough too :D :D Feb 3, 2017 at 8:37
• Please explain it a little. Feb 7, 2017 at 3:25

In Bash:

``````#! /bin/bash

file="YOUR_FILE.txt"

TOTAL=0
do
TOTAL=\$(( TOTAL + line ))
echo \$TOTAL
done <"\$file"
``````
• bash is extremely slow on this: `real 0m53.116s`, almost a minute, on 1.3 million lines :) Feb 3, 2017 at 6:22
• @JacobVlijm dash is about twice as fast, busybox ash and zsh (in sh mode) 1.5 times, but of course, even dash is 5 times slower than python.
– muru
Feb 3, 2017 at 9:04

To print partial sums of integers given on the standard input one per line:

``````#!/usr/bin/env python3
import sys

partial_sum = 0
for n in map(int, sys.stdin):
partial_sum += n
print(partial_sum)
``````

If for some reason the command is too slow; you could use the C program:

``````#include <stdint.h>
#include <ctype.h>
#include <stdio.h>

int main(void)
{
uintmax_t cumsum = 0, n = 0;
for (int c = EOF; (c = getchar()) != EOF; ) {
if (isdigit(c))
n = n * 10 + (c - '0');
else if (n) { // complete number
cumsum += n;
printf("%ju\n", cumsum);
n = 0;
}
}
if (n)
printf("%ju\n", cumsum + n);
return feof(stdin) ? 0 : 1;
}
``````

To build it and run, type:

``````\$ cc cumsum.c -o cumsum
\$ ./cumsum < input > output
``````

`UINTMAX_MAX` is `18446744073709551615`.

The C code is several times faster than the awk command on my machine for the input file generated by:

``````#!/usr/bin/env python3
import numpy.random
print(*numpy.random.random_integers(100, size=2000000), sep='\n')
``````

You probably want something like this:

``````sort -n <filename> | uniq -c | awk 'BEGIN{print "Number\tFrequency"}{print \$2"\t"\$1}'
``````

Explanation of the command:

• `sort -n <filename> | uniq -c` sorts the input and returns a frequency table
• `| awk 'BEGIN{print "Number\tFrequency"}{print \$2"\t"\$1}'` turns the ooutput into a nicer Format

Example:
Input File `list.txt`:

``````4
5
3
4
4
2
3
4
5
``````

The command:

``````\$ sort -n list.txt | uniq -c | awk 'BEGIN{print "Number\tFrequency"}{print \$2"\t"\$1}'
Number  Frequency
2   1
3   2
4   4
5   2
``````
• I like this the out put is nice :)... Feb 2, 2017 at 8:57

You can do this in vim. Open the file and type the following keystrokes:

``````qaqqayiwj@"<C-a>@aq@a:wq<cr>
``````

Note that `<C-a>` is actually ctrl-a, and `<cr>` is carriage return, i.e. the enter button.

Here's how this works. First off, we want to clear out register 'a' so that it has no side-effects on the first time through. This is simply `qaq`. Then we do the following:

``````qa                  " Start recording keystrokes into register 'a'
yiw               " Yank this current number
j              " Move down one line. This will break the loop on the last line
@"            " Run the number we yanked as if it was typed, and then
<C-a>       " increment the number under the cursor *n* times
@a     " Call macro 'a'. While recording this will do nothing
q    " Stop recording
@a  " Call macro 'a', which will call itself creating a loop
``````

After this recursive macro is done running, we simply call `:wq<cr>` to save and quit.

• +1 for breaking down the magic incantation and explaining all the parts. Far too rare round these parts. Feb 3, 2017 at 15:00

Perl one-liner:

``````\$ perl -lne 'print \$sum+=\$_' input.txt
3
7
12
20
``````

With 2.5 million lines of numbers, it takes about 6.6 seconds to process:

``````\$ time perl -lne 'print \$sum+=\$_' large_input.txt > output.txt
0m06.64s real     0m05.42s user     0m00.09s system

\$ wc -l large_input.txt
2500000 large_input.txt
``````
• `real 0m0.908s`, quite nice. Feb 2, 2017 at 21:36
• @JacobVlijm that's on a pretty small file. I added a small test with 2.5 million lines file. 6.64 seconds Feb 2, 2017 at 21:46
• I ran 1.3 million lines on an ancient system Feb 2, 2017 at 21:47

A simple Bash one-liner:

``````x=0 ; while read n ; do x=\$((x+n)) ; echo \$x ; done < INPUT_FILE
``````

`x` is the cumulated sum of all numbers from the current line and above.
`n` is the number in the current line.

We loop over all the lines `n` of `INPUT_FILE` and add their numeric value to our variable `x` and print that sum during each iteration.

Bash is a bit slow here though, you can expect this to run around 20-30 seconds for a file with 2 million entries, without printing the output to the console (which is even slower, independend of the method you use).

Similar to @steeldriver's answer, but with the slightly less arcane `bc` instead:

``````sed 's/.*/a+=&;a/' input | bc
``````

The nice thing about `bc` (and `dc`) is that they are arbitrary precision calculators, so will never overflow or suffer lack of precision over integers.

The `sed` expression transforms the input to:

``````a+=3;a
a+=4;a
a+=5;a
a+=8;a
``````

This is then evaluated by `bc`. The `a` bc variable is auto-initialised to 0. Each line increments `a`, then explicitly prints it.

• `real 0m5.642s` on 1.3 million lines. sed is really slow on this. Feb 3, 2017 at 6:17