7

I have a log file with this log format:

###<Aug 8, 2016 11:59:05 PM>
different text
...
different text
###<Aug 15, 2016 9:10:55 AM>
different text
...
...
...
different text
###<Aug 22, 2016 10:02:17 PM>
different text
...
...
...
...
different text
###<Sep 1, 2016 1:00:01 AM>
different text
###<Sep 7, 2016 3:00:01 PM>
different text
...
...
different text

How do I split this log file by date to files YYYY_MM_DD.log?

  • 3
    maybe logrotate helps you, it is fairly easy to use. – Armita Sep 19 '16 at 11:29
  • Use the axe command. – oarfish Sep 19 '16 at 14:43
7

A perl solution, taking advantage of GNU date to convert the dates:

perl -ne 'if(/^###<(.*)>/){
            chomp($d=`date -d \"$1\" +%Y_%m_%d`);
            $name="$d.log"
          } 
          open(my $fh,">>","$name"); 
          print $fh $_;' file.log 

Explanation

  • -ne : read the input file line by line (saving each line as the special variable $_) and apply the script given by -e to each line.
  • if(/^###<(.*)>/) : if the line starts with ###<, capture everything between the <> as $1 (that's what the parentheses do).
  • chomp($d=date -d \"$1\" +%Y_%m_%d); : the date command reformats the date. For example:

    $ date -d "Sep 1, 2016 1:00:01 AM" +%Y_%m_%d
    2016_09_01
    

    The chomp removes the final newline from the result of date so we can use it later.

  • $name="$d.log" : we save the result of the date command plus .log as the variable $name.
  • open(my $fh,">>","$name"); : open the file $name as the file handle $fh. Don't worry if you don't know what a file handle is, this just means that print $fh "foo" will print foo into $name.
  • print $fh $_; : print the current line into the file that the file handle $fh points to. So, print the line into whatever is currently saved as $name.
6

One approach for solving this could be to use awk. For example, this command:

awk -F'[ <,]+' '/^###/{close(f);f=$4"_"$2"_"$3".log"}{print >> f}END{close(f)}' file

should split the file into the files, using the date fields as filenames

  • Nice! Note that this will create files called 2016_Sep_1.log instead of 2016_09_01 which is what I think the OP asked for. – terdon Sep 19 '16 at 12:17
  • @terdon: That is correct. I left it like this for simplicity, but the answer could be adapted to use the date command like in the other two answers. – user000001 Sep 19 '16 at 12:19
6

With awk:

awk '/^#+<[^>]+>$/ {if (lines) print lines >file; \
     dt=gensub("^#+<([^>]+)>$", "\\1", $0)
     dt_cmd="date -d \""dt"\" +%Y_%m_%d.log" \
     dt_cmd | getline file; lines=$0; next}; \
     {lines=lines ORS $0} END {print lines >file}' file.log

Readable form:

awk '
      /^#+<[^>]+>$/ {
                    if (lines) 
                        print lines >file
                    dt=gensub("^#+<([^>]+)>$", "\\1", $0)
                    dt_cmd="date -d \""dt"\" +%Y_%m_%d.log"
                    dt_cmd | getline file; lines=$0
                    next
                    }
      {
      lines=lines ORS $0
      } 
      END {
          print lines >file
          }' file.log
  • /^#+<[^>]+>$/ matches the lines containing dates, the chunk surrounded by {} will only be run if the condition matches. If matches, we are getting the date in desired format by using external date command and saving the output in variable file, and saving the content of variable lines so far as file file (from previous chunk), and then instantiate the variable lines again with the line

  • For all other lines, we concatenating the lines as variable lines

  • The last chunk is saved by putting in the END block

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.