Use `less` to view BIG (multi-gigabyte) CSV files [closed]

Question

I have a big (multi-gigabyte) CSV file, which I am viewing with a puny computer.

I would like to view my file using less or some command like it which doesn't have to read the whole file at once to show me part of it.

Is there a command out there which can show me my file in comma-aligned columns?

This question appears to be abandoned, if you are experiencing a similar issue please ask a new question with details pertaining to your problem. If you feel this question is not abandoned, please flag the question explaining that. :)

fossfreedom · Answer 1 · 2011-08-16 20:12:57Z

up vote 0 down vote

There is a csv column formatting question on another StackOverflow site - does this work for your very-large file?

cat somefile.csv | column -s, -t | less -#2 -N -S

answered Aug 16 '11 at 20:12

fossfreedom♦
75.8k11161209

	This will not work. Because cat will try to print the entire file, which may be gigabytes worth of data. It may be that there's a buffer somewhere in the pipes which prevents this situation, but I'm not sure and a complete answer using `cat` would have to address this. If there were such a buffer, I'd begin to wonder if the data was then being cached by `less`. – Richard Aug 16 '11 at 20:21
	ok - please add this comment to your question - it will give a better understanding that you are manipulating big=multi gigabytes!! – fossfreedom♦ Aug 16 '11 at 20:23
	I'd assumed that big was a sufficient descriptor! I'll make a note of that, thanks. – Richard Aug 16 '11 at 20:27
	Can you use `tail` or `head` instead of `cat` to get the last 10 or first 10 records, then pipe that into `column`? `head file.csv \| column <options> \| less <options>`. I am not sure if `head` or `tail` will read the entire file THEN output the desired rows... – SaultDon Aug 16 '11 at 20:39

queueoverflow · Answer 2 · 2011-08-16 22:15:40Z

I just tried vim with a 514M text file. It has 8032416 lines and after it loaded for around 30 seconds, I could run around the file very smoothly.

It took 30% of my 2GB RAM, and vim created a 2.2M swap file. Sounds reasonable.

tail gave me instant results on that file, it probably seeks to the bottom and then goes up until it reaches 10 lines and then prints them.

closed as too localized by jrg♦ Feb 23 '12 at 21:02