11

I have a text file with 2 million lines. Each line has a positive integer. I am trying to form a frequency table kind of thing.

Input file:

3
4
5
8

Output should be:

3
7
12
20

How do I go about doing this?

6
  • 1
    In your text, you say, that you want a frequency table. Your output sample is a list. Can you please clarify this?
    – Wayne_Yux
    Feb 2, 2017 at 8:34
  • Indeed your output is not a frequency table
    – don.joey
    Feb 2, 2017 at 8:41
  • I'm sorry. I meant a cumulative frequency table. Have modified the question. Thanks.
    – user590995
    Feb 2, 2017 at 10:18
  • It's not very cool but I usually just do stuff like this into a spreadsheet.
    – John U
    Feb 3, 2017 at 15:05
  • @JohnU I usually do, but the file I have has 1 million numbers.
    – user590995
    Feb 6, 2017 at 9:34

10 Answers 10

21

With awk:

awk '{total += $0; $0 = total}1'

$0 is the current line. So, for each line, I add it to the total, set the line to the new total, and then the trailing 1 is an awk shortcut - it prints the current line for every true condition, and 1 as a condition evaluates to true.

9
  • Please could you explain your code? Feb 2, 2017 at 8:43
  • Can the word print be used also? Feb 2, 2017 at 8:45
  • Yes, print total}, instead of $0 = total}1
    – muru
    Feb 2, 2017 at 8:47
  • 1
    @George ah, no.
    – muru
    Feb 2, 2017 at 8:53
  • 10
    A shorter and perhaps more understandable way of writing the awk script would be {print(total += $0)}
    – Miles
    Feb 4, 2017 at 3:25
9

In a python script:

#!/usr/bin/env python3
import sys

f = sys.argv[1]; out = sys.argv[2]

n = 0

with open(out, "wt") as wr:
    with open(f) as read:
        for l in read:
            n = n + int(l); wr.write(str(n)+"\n")

To use

  • Copy the script into an empty file, save it as add_last.py
  • Run it with the source file and targeted output file as arguments:

    python3 /path/to/add_last.py <input_file> <output_file>
    

Explanation

The code is rather readable, but in detail:

  • Open output file for writing results

    with open(out, "wt") as wr:
    
  • Open input file for reading per line

    with open(f) as read:
        for l in read:
    
  • Read the lines, adding the value of the new line to the total:

    n = n + int(l)
    
  • Write the result to the output file:

    wr.write(str(n)+"\n")
    
11
  • 2
    The code can be simplified
    – jfs
    Feb 2, 2017 at 20:33
  • 3
    It is not about shortness or time performance (million lines is not big data). The code in your answer is not idiomatic Python. My answer is just more pythonic version of yours.
    – jfs
    Feb 2, 2017 at 21:16
  • 8
    @J.F.Sebastian if the more idiomatic version is slower why would anyone prefer it? There's nothing special about being "pythonic" that's just a convention that helps python devs share code and standards for readability. If the more idiomatic version is less efficient (slower) then it shouldn't be used unless you're working in an environment where standardization is more important than performance (which sounds like a horrible idea to me).
    – terdon
    Feb 3, 2017 at 8:55
  • 2
    @terdon there is something to be said about premature optimisation. Readability can be important because of long-term maintainability.
    – muru
    Feb 3, 2017 at 9:04
  • 4
    @muru sure, but this is perfectly readable. It's only crime is not being "pythonic". Not to mention that we're talking about 7 lines of code, not some giant project. Sacrificing efficiency in the name of style conventions seems like the wrong approach.
    – terdon
    Feb 3, 2017 at 9:10
9

Just for fun

$ sed 'a+p' file | dc -e0 -
3
7
12
20

This works by appending +p to each line of the input, and then passing the result to the dc calculator where

   +      Pops two values off the stack, adds them, and pushes the result.
          The precision of the result is determined only by the values  of
          the arguments, and is enough to be exact.

then

   p      Prints  the  value on the top of the stack, without altering the
          stack.  A newline is printed after the value.

The -e0 argument pushes 0 onto the dc stack to initialize the sum.

4
  • Something like this might actually be the fastest over a large dataset Feb 3, 2017 at 2:26
  • @DigitalTrauma on a 1.3 million lines, actually almost the slowest: real 0m4.234s Feb 3, 2017 at 6:14
  • fun is all it needs for an upvote :D quirky is enough too :D :D
    – Rinzwind
    Feb 3, 2017 at 8:37
  • Please explain it a little.
    – AmanicA
    Feb 7, 2017 at 3:25
8

In Bash:

#! /bin/bash

file="YOUR_FILE.txt"

TOTAL=0
while IFS= read -r line
do
    TOTAL=$(( TOTAL + line ))
    echo $TOTAL
done <"$file"
2
  • bash is extremely slow on this: real 0m53.116s, almost a minute, on 1.3 million lines :) Feb 3, 2017 at 6:22
  • @JacobVlijm dash is about twice as fast, busybox ash and zsh (in sh mode) 1.5 times, but of course, even dash is 5 times slower than python.
    – muru
    Feb 3, 2017 at 9:04
6

To print partial sums of integers given on the standard input one per line:

#!/usr/bin/env python3
import sys

partial_sum = 0
for n in map(int, sys.stdin):
    partial_sum += n
    print(partial_sum)

Runnable example.

If for some reason the command is too slow; you could use the C program:

#include <stdint.h>
#include <ctype.h>
#include <stdio.h>

int main(void)
{
  uintmax_t cumsum = 0, n = 0;
  for (int c = EOF; (c = getchar()) != EOF; ) {
    if (isdigit(c))
      n = n * 10 + (c - '0');
    else if (n) { // complete number
      cumsum += n;
      printf("%ju\n", cumsum);
      n = 0;
    }
  }
  if (n)
    printf("%ju\n", cumsum + n);
  return feof(stdin) ? 0 : 1;
}

To build it and run, type:

$ cc cumsum.c -o cumsum
$ ./cumsum < input > output

Runnable example.

UINTMAX_MAX is 18446744073709551615.

The C code is several times faster than the awk command on my machine for the input file generated by:

#!/usr/bin/env python3
import numpy.random
print(*numpy.random.random_integers(100, size=2000000), sep='\n')
1
5

You probably want something like this:

sort -n <filename> | uniq -c | awk 'BEGIN{print "Number\tFrequency"}{print $2"\t"$1}'

Explanation of the command:

  • sort -n <filename> | uniq -c sorts the input and returns a frequency table
  • | awk 'BEGIN{print "Number\tFrequency"}{print $2"\t"$1}' turns the ooutput into a nicer Format

Example:
Input File list.txt:

4
5
3
4
4
2
3
4
5

The command:

$ sort -n list.txt | uniq -c | awk 'BEGIN{print "Number\tFrequency"}{print $2"\t"$1}'
Number  Frequency
2   1
3   2
4   4
5   2
1
  • I like this the out put is nice :)... Feb 2, 2017 at 8:57
5

You can do this in vim. Open the file and type the following keystrokes:

qaqqayiwj@"<C-a>@aq@a:wq<cr>

Note that <C-a> is actually ctrl-a, and <cr> is carriage return, i.e. the enter button.

Here's how this works. First off, we want to clear out register 'a' so that it has no side-effects on the first time through. This is simply qaq. Then we do the following:

qa                  " Start recording keystrokes into register 'a'
  yiw               " Yank this current number
     j              " Move down one line. This will break the loop on the last line
      @"            " Run the number we yanked as if it was typed, and then
        <C-a>       " increment the number under the cursor *n* times
             @a     " Call macro 'a'. While recording this will do nothing
               q    " Stop recording
                @a  " Call macro 'a', which will call itself creating a loop

After this recursive macro is done running, we simply call :wq<cr> to save and quit.

1
  • 1
    +1 for breaking down the magic incantation and explaining all the parts. Far too rare round these parts.
    – John U
    Feb 3, 2017 at 15:00
5

Perl one-liner:

$ perl -lne 'print $sum+=$_' input.txt                                                                
3
7
12
20

With 2.5 million lines of numbers, it takes about 6.6 seconds to process:

$ time perl -lne 'print $sum+=$_' large_input.txt > output.txt                                        
    0m06.64s real     0m05.42s user     0m00.09s system

$ wc -l large_input.txt
2500000 large_input.txt
3
  • real 0m0.908s, quite nice. Feb 2, 2017 at 21:36
  • @JacobVlijm that's on a pretty small file. I added a small test with 2.5 million lines file. 6.64 seconds Feb 2, 2017 at 21:46
  • 1
    I ran 1.3 million lines on an ancient system Feb 2, 2017 at 21:47
3

A simple Bash one-liner:

x=0 ; while read n ; do x=$((x+n)) ; echo $x ; done < INPUT_FILE

x is the cumulated sum of all numbers from the current line and above.
n is the number in the current line.

We loop over all the lines n of INPUT_FILE and add their numeric value to our variable x and print that sum during each iteration.

Bash is a bit slow here though, you can expect this to run around 20-30 seconds for a file with 2 million entries, without printing the output to the console (which is even slower, independend of the method you use).

3

Similar to @steeldriver's answer, but with the slightly less arcane bc instead:

sed 's/.*/a+=&;a/' input | bc

The nice thing about bc (and dc) is that they are arbitrary precision calculators, so will never overflow or suffer lack of precision over integers.

The sed expression transforms the input to:

a+=3;a
a+=4;a
a+=5;a
a+=8;a

This is then evaluated by bc. The a bc variable is auto-initialised to 0. Each line increments a, then explicitly prints it.

1
  • real 0m5.642s on 1.3 million lines. sed is really slow on this. Feb 3, 2017 at 6:17

You must log in to answer this question.