I want to split a text file according to a pre-defined set of rows. For example. I have a file


And then I have the following sets of rows (these could be stored however it is more convenient, on one file, multiple files,...).


I want to split my file so that I get 3 files back like:







Here is a bash script assuming your input file is named infile and the ranges are stored 1-per-line in a file named splits:

for range in $(< splits); do
  sed -n "$(echo "$range" | cut -f1 -d, )p" infile > "file$i"
  sed -n "$(echo "$range" | cut -f2 -d, )p" infile >> "file$i"

This simply uses sed to print the lines specified by the ranges, and saves each result as a new file (files created are named file1 file2 file3 etc). Two invocations of sed are used to preserve the specified order of the rows.

Note that there is no format or error checking done by this simple script, and existing files named e.g. file1 will be overwritten.


A simplified alternative (courtesy of @muru) using while read and letting bash split the ranges instead of cut:

while IFS=',' read n1 n2 
    sed -n "$n1 p; $n2 p" infile > "file$i"
done < splits

If the order of the lines in the output files is important (e.g. rows 5,4 != 4,5), then the sed bit will need to be broken up into two separate invocations similar to the first script.


The following python script will do the split:


import argparse

parser = argparse.ArgumentParser()
parser.add_argument('file', type=argparse.FileType('r'))
parser.add_argument('lines', type=argparse.FileType('r'))

args = parser.parse_args()

file_lines = list(args.file)

for i, l in enumerate(args.lines):
    r = l.rstrip().split(',')
    with open('file{}'.format(i+1), 'w') as f:
        for k in r:
            except IndexError: # Ignore lines out of range

Simply call it this way:

./split.py file lines

Where <file> is the abcdef file and <lines> the 1,2... lines range (you can even have multiple lines like 1,6,3,18,5)


Here's one way to do it, in awk

awk -F, 'NR==FNR {for (i=1;i<=NF;i++) a[$i]=FNR; next;} {print $0 >> "outfile"a[FNR];}' index file

It reads the index file, and saves its line number (FNR) into an array that's indexed by the list of values on the line. Then it reads the input file, and uses its line number to look up what output file number to write each line to.


Another simply solution is :)

awk -F, 'NR==FNR{ X[NR]=$0; next } {print X[$1] RS X[$2]>"out"FNR}' file lines


NR==FNR    - Execute next block for 1st file only (*file*)
X[NR]=$0   - Create an associative array with key as 'NR' (line number) and copy
             whole line ($0) into it as its content.
next       - Jump to reading the next row from *file* (1st file)

print X[$1] RS X[$2]   
           - Print those line from array X that its line-number is the same as 
             value of first field in *lines* file then print a new-line(RS) and 
             print the line that its line-number is the same as value of second 
             field in *lines* file again and redirect the result into out#

Another bash solution, assuming input as the input, pattern as the pattern and output as the output:

i=0 # set the output number to 0
while read row; do # for each line in file `pattern` as $row
    columns=$(<<< $row tr ',' '\n') # store each line obtained by transforming ',' in '\n' inside $row in an array $columns
    for column in $columns; do # for each member in array $columns as $column
        sed -n "${column}p" input
    done > output$i # write column $column in `input` to `output$i`
    ((i++)) # increment the output number
done < pattern
  • @terdon Don't know how I could miss that, thanks – kos Mar 20 '15 at 6:25

The python script below will do the job as well:

#!/usr/bin/env python3
#--- set the paths below
f1 = "/path/to/predifined_rows.txt"; f2 = "/path/to/outtakes.txt"; save_to = "directory/to/save/the/outtakes"

rows = [l.strip() for l in open(f1).readlines()]
outtakes = [eval(l.strip()) for l in open(f2).readlines()]
for i in range(len(outtakes)):
    s = ("\n").join([rows[n-1] for n in outtakes[i]])
    with open(save_to+"/"+str(i+1), "wt") as out:
  • Copy it into an empty file, save it as outtake.py
  • In the headsection, set the path(s) to f1, f2 and the directory to save the files to
  • Run it with the command

    python3 /path/to/outtake.py

What it does

  1. it reads the numbers from the second file, reading the lines as a list of numbers
  2. for each of the rows, it collects the corresponding items of the first file (by index) and writes it out, to separately numbered files, in the directory defined in save_to

You can use the tool split. A bunch of examples can be found e.g. here

However, in your case something like

split -l 2 <inputfile>

will create a set of files with two lines called something like xaa, xab ...

  • 1
    Oh... I see my answer is wrong. AFAIK split cannot reorder your file. – frlan Mar 19 '15 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.