Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

Is there an application or a command that could to split a csv (preserving the a table header) into smaller equal parts?

I'm trying to import a large 36mb file and the importer is going kaputz.

Here's an example of a windows application: http://download.fyxm.net/CSV-Splitter-33994.html

I guess i could use wine to use this program but would prefer something native of ubuntu.

share|improve this question
2  
Could explain better, preferably with an example? –  enzotib Aug 25 '11 at 14:46

4 Answers 4

up vote 7 down vote accepted

You can use csvfix.

It's a command line tool for managing CSV files. The download page doesn't have any Linux binaries, but you can compile the source code from a makefile.

Here is a list of commands. In particular, the file_split command looks to be what you're after.

The manual for csvfix can be found here.

share|improve this answer
    
How would you go ahead an compile this? –  chrisjlee Aug 25 '11 at 21:52
2  
Download the version 1.1 source code here. Extract the zip archive. Open a terminal and cd to the extracted directory. Run make lin to compile the program. The csvfix binary will be in csvfix-build/csvfix/bin. You can run it from that directory with ./csvfix or copy it to /usr/local/bin where you'll have access to it anywhere. –  Kris Harper Aug 25 '11 at 22:47
    
Thank you! I really appreciate your help. –  chrisjlee Aug 26 '11 at 14:00

I also made an online version that allows splitting by lines, file size, or file count. It's quite intuitive and user-friendly.

CSV Live Splitter

share|improve this answer

I made one because I wanted an online one I could use on my mac..

check this

share|improve this answer

So you want to have multiple files which contain part of the entries in the main file and each contains the table header?

If that is what you want, one could write a simple script that takes the first file and the number of files that you want (or the number of lines in each file) and creates those files.

If it does not need to be graphical and you tell me what parameters you want to set, I can try to make such a script for you.


This is my quick and dirty solution:

#!/usr/bin/python

import optparse
import sys
import os

parser = optparse.OptionParser(usage="csv-splitter infile", description="Takes a CSV file and splits it into smaller files. The header of the infile is written to each file.")
parser.add_option("-p", "--parts", dest="parts", type="int", default=3, help="number of resulting files")

(options, args) = parser.parse_args()
del parser

infile = args[0]

lines = 0

if not os.path.exists(infile):
    print "file does not exist"
    sys.exit(1)

print "counting lines ..."
with open(infile) as h:
    for line in h:
        lines += 1

lines_per_file = lines/options.parts
print "found %d lines, that makes %d lines per file" % (lines, lines_per_file)

with open(infile) as h:
    header = h.readline()
    read = h.readline()
    for filenumber in xrange(options.parts):
        written_lines = 0
        print "writing to file %d of %d ..." % (filenumber+1, options.parts)
        with open(infile+"-part-"+str(filenumber)+".csv", "w") as outfile:
            outfile.write(header)
            while read != '' and (written_lines <= lines_per_file or filenumber+1 == options.parts):
                outfile.write(read)
                written_lines += 1
                read = h.readline()

print "done"
share|improve this answer
    
Yes that's what i'm looking for. –  chrisjlee Aug 25 '11 at 14:52
    
Which option do you need, how many files or how many lines per file? Is command line okay? –  queueoverflow Aug 25 '11 at 14:54
    
This 1 file should be split into 3 parts (currently 36mb). It's 70,000 lines. So hopefully each file is around 10mb. –  chrisjlee Aug 25 '11 at 15:00
    
Okay, I'll see what I can do. –  queueoverflow Aug 25 '11 at 15:08
    
no need to! I think i found something someone wrote already. See the answer above. –  chrisjlee Aug 25 '11 at 15:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.