Select the remaining text in a line from a log file

Question

I'm using Bash shell to process some log files, which I need to put in a CSV file. The information I need is process start date and time, process end date and time , Process ID and message

What I did, grep lines including started process and put them in a file, then Grep ended processes and put them in a second file. Once done, I take each file, and use awk to split and extract the information needed, here is a peace of code I'm using,

input=starts.txt
while IFS= read -r line
do
    procs=`echo $line | awk  '{ print $6;}' 
    date_s=`echo $line | awk '{ print $1;}'`
    time_s=`echo $line |  awk '{ print $2;}'`
    m1=`echo $line |  awk '{ print $3;}'`
    m2=`echo $line |  awk '{ print $4;}'`
    m3=`echo $line |  awk '{ print $5;}'`
    m4=`echo $line |  awk '{ print $7;}'`
    m5=`echo $line |  awk '{ print $8;}'`
    m6=`echo $line |  awk '{ print $9;}'`

 echo $procs ";" $date_s ";" $time_s ";" $m1 $m2  $m3 $m4 $m5 $m6 

one   < "$input" > result.csv

The lines have the following format :

02/01/2018 10:32:35      ANR4930I Reclamation process 1320 started for primary      storage pool VM_VTL_POOL automatically, threshold=75,     duration=None. (PROCESS: 1320) 

I have two issues now :

1. The while loop doesn't end.
2. After extracting the time/date and process ID, I want to put the remaining message in a separate field, without taking word by word and concatenating them (m1 m2 m 3 ...) Also, if there is any improvements that can be done to my code.

Answer 1

The biggest improvement will be to avoid the shell loop altogether and process each record (line) directly in Awk. For example:

$ awk '{
    printf("%s;%s;%s;", $6, $1, $2)
    for(i=3; i<NF;i++) {
      if(i==6) continue; 
      printf("%s ", $i)
    }
    printf("%s\n",$NF)
  }' input
1320;02/01/2018;10:32:35;ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320)

Answer 2

I read your task as "move the field 5 (process number) to the front, and then output the first 3 fields separated by ;, followed by ; and then the remainder.

In Perl I'd do this like so (as a one-liner):

perl -a -n -l -e \
    'unshift @F, splice(@F, 5, 1); 
     print join(";", @F[0..2]), ";@F[3..$#F]";' \
    < input.txt > output.csv

Result:

input.txt:

02/01/2018 10:32:35      ANR4930I Reclamation process 1320 started for primary      storage pool VM_VTL_POOL automatically, threshold=75,     duration=None. (PROCESS: 1320) 
02/01/2018 10:32:35      ANR4930I Reclamation process 4567 started for primary      storage pool VM_VTL_POOL automatically, threshold=75,     duration=None. (PROCESS: 1320) 

output.csv:

1320;02/01/2018;10:32:35;ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320)
4567;02/01/2018;10:32:35;ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320)

---

Explanation:

perl -a -n -l -e

• split each input line at whitespace and put the result into the predefined array @F
• process each input line (but don't print it yet)
• use the input separator (\n) also as the output separator (quite simplified)
• run the following expression for each input line

unshift @F, splice(@F, 5, 1);

• removes the 5^th element (count=1) from the array @F (which contains one line of your input file split at whitespace) and prepends that 5^th element in front of the array @F.

print join(";", @F[0..2]), ";@F[3..$#F]";'

• prints the first 2 elements of @F with ; in-between, then
• followed by a literal ; and then the rest of the array @F starting at the 3^rd element up to the end with a space as a separator. (print "@any_array" prints the items separated by a space.)

---

If you want the same code as a script (say format-messages.pl), then it looks slighly different because the command line switches to Perl (which are now missing) implicitely add some code which now has to be added explicitely. (Yes, there are other ways, but ...)

#!/usr/bin/env perl

use strict;
use warnings;

while(<>) {
    my @F = split;
    unshift @F, splice(@F, 5, 1); 
    print join(";", @F[0..2]), ";@F[3..$#F]\n";
}

Do a chmod +x format-messages.pl and then run this script with ./format-messages.pl < input.txt > output.csv