2

I'm using Bash shell to process some log files, which I need to put in a CSV file. The information I need is process start date and time, process end date and time , Process ID and message

What I did, grep lines including started process and put them in a file, then Grep ended processes and put them in a second file. Once done, I take each file, and use awk to split and extract the information needed, here is a peace of code I'm using,

input=starts.txt
while IFS= read -r line
do
    procs=`echo $line | awk  '{ print $6;}' 
    date_s=`echo $line | awk '{ print $1;}'`
    time_s=`echo $line |  awk '{ print $2;}'`
    m1=`echo $line |  awk '{ print $3;}'`
    m2=`echo $line |  awk '{ print $4;}'`
    m3=`echo $line |  awk '{ print $5;}'`
    m4=`echo $line |  awk '{ print $7;}'`
    m5=`echo $line |  awk '{ print $8;}'`
    m6=`echo $line |  awk '{ print $9;}'`

 echo $procs ";" $date_s ";" $time_s ";" $m1 $m2  $m3 $m4 $m5 $m6 

one   < "$input" > result.csv

The lines have the following format :

02/01/2018 10:32:35      ANR4930I Reclamation process 1320 started for primary      storage pool VM_VTL_POOL automatically, threshold=75,     duration=None. (PROCESS: 1320) 

I have two issues now :

  1. The while loop doesn't end.
  2. After extracting the time/date and process ID, I want to put the remaining message in a separate field, without taking word by word and concatenating them (m1 m2 m 3 ...) Also, if there is any improvements that can be done to my code.
4

I read your task as "move the field 5 (process number) to the front, and then output the first 3 fields separated by ;, followed by ; and then the remainder.

In Perl I'd do this like so (as a one-liner):

perl -a -n -l -e \
    'unshift @F, splice(@F, 5, 1); 
     print join(";", @F[0..2]), ";@F[3..$#F]";' \
    < input.txt > output.csv

Result:

input.txt:

02/01/2018 10:32:35      ANR4930I Reclamation process 1320 started for primary      storage pool VM_VTL_POOL automatically, threshold=75,     duration=None. (PROCESS: 1320) 
02/01/2018 10:32:35      ANR4930I Reclamation process 4567 started for primary      storage pool VM_VTL_POOL automatically, threshold=75,     duration=None. (PROCESS: 1320) 

output.csv:

1320;02/01/2018;10:32:35;ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320)
4567;02/01/2018;10:32:35;ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320)

Explanation:

perl -a -n -l -e

  • split each input line at whitespace and put the result into the predefined array @F
  • process each input line (but don't print it yet)
  • use the input separator (\n) also as the output separator (quite simplified)
  • run the following expression for each input line

unshift @F, splice(@F, 5, 1);

  • removes the 5th element (count=1) from the array @F (which contains one line of your input file split at whitespace) and prepends that 5th element in front of the array @F.

print join(";", @F[0..2]), ";@F[3..$#F]";'

  • prints the first 2 elements of @F with ; in-between, then
  • followed by a literal ; and then the rest of the array @F starting at the 3rd element up to the end with a space as a separator. (print "@any_array" prints the items separated by a space.)

If you want the same code as a script (say format-messages.pl), then it looks slighly different because the command line switches to Perl (which are now missing) implicitely add some code which now has to be added explicitely. (Yes, there are other ways, but ...)

#!/usr/bin/env perl

use strict;
use warnings;

while(<>) {
    my @F = split;
    unshift @F, splice(@F, 5, 1); 
    print join(";", @F[0..2]), ";@F[3..$#F]\n";
}

Do a chmod +x format-messages.pl and then run this script with ./format-messages.pl < input.txt > output.csv

|improve this answer|||||
  • You are right, that's exactly what I'm trying to do using Bash shell. Thanks – Aziz Azizos Jul 1 '18 at 15:33
  • @AzizAzizos Good to hear I understood it correctly. But it sounds you want a strict bash solution, right? If so, then I'm afraid you will need to wait for another answer... Never mind, though. – PerlDuck Jul 1 '18 at 15:42
  • That's right, I tested your script and it's giving the correct results, so just need to convert it to Bash script :) Thanks again. – Aziz Azizos Jul 1 '18 at 15:45
  • @steeldriver I'm not exactly a bash hero but I reckon it is not that difficult to split a line into an array at whitespace, swap two array items, and then print the first few with ; and the remainder with ` ` as a separator in bash. My solution also uses a loop, albeit it's hidden. – PerlDuck Jul 1 '18 at 16:10
  • Thank you guys, I never used perl before, is there a simple way to put this code into a script file and run it, something like perl my_perl_scpt.pl input.txt will generate result.csv – Aziz Azizos Jul 1 '18 at 16:10
5

The biggest improvement will be to avoid the shell loop altogether and process each record (line) directly in Awk. For example:

$ awk '{
    printf("%s;%s;%s;", $6, $1, $2)
    for(i=3; i<NF;i++) {
      if(i==6) continue; 
      printf("%s ", $i)
    }
    printf("%s\n",$NF)
  }' input
1320;02/01/2018;10:32:35;ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320)
|improve this answer|||||
  • The last part of the line is still missing ** duration=None. (PROCESS: 1320) ** not sure if this is because of the comma character – Aziz Azizos Jul 1 '18 at 15:01
  • 1
    Well, you didn't provide an example of your desired output - so I based it on the fields from your script. Please see updated answer. – steeldriver Jul 1 '18 at 15:04
  • Sorry :) the desired output is : 1320; 02/01/2018 ; 10:32:35; ANR4930I Reclamation process started for primary storage pool VM_VTL_POOL automatically, threshold=75, duration=None. (PROCESS: 1320) – Aziz Azizos Jul 1 '18 at 15:12
  • This gives the very same output as the Perl solution. So it cannot be wrong ;-) (kidding, but +1 anyway). – PerlDuck Jul 1 '18 at 16:31
  • Bash can read and split fields directly into an array with read -a [variable]. The field delimiter(s) is/are taken from $IFS as usually. – David Foerster Jul 1 '18 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.