Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.


I'm trying to analyze an enormous text file (1.6GB), whose data lines look like this:

20090118025859 -2.400000 78.100000 1023.200000 0.000000
20090118025900 -2.500000 78.100000 1023.200000 0.000000
20090118025901 -2.400000 78.100000 1023.200000 0.000000

I don't even know how many lines there are. But I'm trying to split the file by date. The left number is a time stamp (these lines are from 2009, January 18th).
How can I split this file into pieces according to the date?

Everything I know would be to grep file '20090118*' > data20090118.dat , but there sure is a way to do all the dates at once, right?

The number of entries per date differ, so using split with a constant number won't work.

Thanks in advance,
Alex

share|improve this question
4  
I think this question belongs to StackOverflow. –  Octavian Damiean Mar 2 '11 at 12:02
    
For counting the number of lines, use wc: wc -l file. –  Lekensteyn Mar 2 '11 at 14:41
add comment

3 Answers 3

Assuming the file is sorted and the dates are always there, this should work:

#!/bin/bash

base_dir='./'    

while read line; do
    date="${line:0:8}"
    echo "$line" >> "$base_dir$date.txt"
done < "$1"

[Save it as my_splitter, make it executable by running chmod +x my_splitter, then call it like ./my_splitter input_file]

It reads the input file line by line, extracts the date and uses that to append the lines with the same date to the same file.

base_dir is the target directory, and the files will be of the form <date>.txt. Note: existing files won't be overwritten, new lines would be appended due to the >> redirector, so better make sure the target directory doesn't contain any files of the form <date>.txt.

share|improve this answer
1  
Instead of $(echo "$line" | cut -b 1-8) you can do ${line:0:8}, which is shorter and more efficient. mywiki.wooledge.org/BashSheet#Parameter_Operations –  geirha Mar 2 '11 at 22:07
    
@geirha Thanks, edited. –  htorque Mar 2 '11 at 22:14
add comment

This could probably work for you:

awk '{d=substr($1, 1, 8); fn = "data" d ".dat"; print $0 >> fn}' hugefile
share|improve this answer
    
+1 Nice, I need to learn awk. ;-) –  htorque Mar 2 '11 at 16:16
add comment

I would use {x..y}, maybe for y, m, d cascading, shema:

for d in {18..19} ; do grep 200901$d datadata; echo; done 
20090118025859 -2.400000 78.100000 1023.200000 0.000000
20090118025900 -2.500000 78.100000 1023.200000 0.000000
20090118025901 -2.400000 78.100000 1023.200000 0.000000

20090119025859 -2.400000 78.100000 1023.200000 0.000000
20090119025900 -2.500000 78.100000 1023.200000 0.000000
20090119025901 -2.400000 78.100000 1023.200000 0.000000
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.