4

I have a log file which is output by a script, the log file is rotated daily. It will contain the strings

Transfer started at timestamp 

and

Transfer completed successfully at timestamp

repeatedly, as the mentioned transfer will take place hourly. The timestamps will have been previously created with date.

  • I want to capture the last instance of these two strings, and everything in between, into a separate file.
  • If the started string is found near the end of the log file, with no following completed string, I want to capture everything up to EOF and output an error message to say that the end string was not found.

I'm guessing I'll need to use sed or awk but am really inexperienced with them. I want to use the command in a bash script, and understand what each part is doing, so some explanation would be very useful.

An example chunk of log file:

ERROR - Second tech sync failed with rsync error code 255 at Fri May 27 13:50:4$
--------------------------------------------------------------------
After_sync script completed successfully with no errors.
Main script finished at Fri May 27 13:50:43 BST 2016 with PID of 18808.
--------------------------------------------------------------------
Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.
37 approvals pending.
Transfer completed successfully at Fri May 27 14:05:16 BST 2016
--------------------------------------------------------------------
Local repository verification started at Fri May 27 14:35:02 BST 2016
...

The desired output:

Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.
37 approvals pending.
Transfer completed successfully at Fri May 27 14:05:16 BST 2016

However, if the log file was like this:

ERROR - Second tech sync failed with rsync error code 255 at Fri May 27 13:50:4$
--------------------------------------------------------------------
After_sync script completed successfully with no errors.
Main script finished at Fri May 27 13:50:43 BST 2016 with PID of 18808.
--------------------------------------------------------------------
Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.

I would want to output:

Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.
ERROR: transfer not complete by end of log file
11
  • 3
    You know the drill, add an example chunk and your desired output from that :)
    – heemayl
    Jun 14, 2016 at 14:57
  • Yes, awk can do that, but like heemayl said, we do need sample input and output Jun 14, 2016 at 15:02
  • It's definitely easier to understand with examples @heemayl :)
    – Arronical
    Jun 14, 2016 at 15:13
  • Edited for clarity @Serg
    – Arronical
    Jun 14, 2016 at 15:14
  • I would say the routine you have written in english needs to be broken down into psuedo-code and probably better handled in a proper language like python which can be easier to unit test. Jun 14, 2016 at 15:16

4 Answers 4

2

When I hear "I want to do X with the last something in the file", I think:

  • reverse the file
  • do X with the first something in the file
  • reverse the output of X

In code:

tac logfile | awk '
    BEGIN {text = "ERROR: transfer not complete by end of log file"}
    /^Transfer completed successfully/ {text = ""}
    {text = text ORS $0}
    /^Transfer started at / {print text; exit}
' | tac

Since we are reading the log file from the bottom up, I start with assuming the transfer is not completed. If I see the "transfer completed" message, we can throw out whatever we've captured so far. We save each line. When we see the "transfer started" line, we know we have seen all of the last transfer in the file: print out the (reversed) captured text and exit awk.

2
  • Thanks, I get the tac usage, the only part I can't understand is the {text = text ORS $0} part of the awk command. Are you setting text as the previous content of text then the output record seperator (I'm guessing newline as default) then $0 the line which awk has just read?
    – Arronical
    Jun 14, 2016 at 15:44
  • 1
    Yes, exactly so. awk does not have a "visible" concatenation operator. You just concatenate two strings by putting them side-by-side. This is the concatenation of the previous value of text and a newline and the current line. Jun 14, 2016 at 15:58
2

Just use Python. I don't really have time, but I'd start with this:

#!/usr/bin/env python

start = "Transfer started at"
end = "Transfer completed successfully"
buffer = ""
log = False

for line in open('logfile.log'):
  if line.startswith(start):
    buffer = line
    log = True
  elif line.startswith(end):
    buffer += line
    log = False
  elif log:
    buffer += line

open('output.log', 'w').write(buffer)

if log == True:
  print("End string was not found")
3
  • Thanks for the answer, I'm too old for homework though! I was really looking for a command within a bash script so Python isn't my preferred option. On to learning that next, after getting to grips with more of the bash stuff!
    – Arronical
    Jun 14, 2016 at 15:39
  • @Arronical bash is cool, but kind of unreadable. And this stuff works on any platform with Python, and maybe even on MicroPython. Edited to add error handling if you're not making you homework. =) Jun 18, 2016 at 9:26
1

Python all the things, but let regular expressions do the work for you!

Paste the script below to any file, e.g. logfilter.py and make it executable using the command chmod +x logfilter.py.

Then you may run it like this, assuming it's located in the current directory:

./logfilter.py logfile.txt

This will make it process the file logfile.txt.

However, if you do not pass it any command-line arguments, it will wait for data on standard input. This means you can also pipe data into it. The following example processes data from the clipboard (needs xsel installed to access the clipboard):

xsel -ob | ./logfilter.py

The script:

#! /usr/bin/env python3

p_start = r'^Transfer started at .*?$'
p_end   = r'^Transfer completed successfully at .*?$'

error_no_match = 'ERROR: no match found'
error_no_end   = 'ERROR: transfer not complete by end of log file'

pattern = r'{p0}(?!.*{p0})(?:.*?{p1}|.*)'.format(p0=p_start, p1=p_end)

import sys, re
if len(sys.argv) > 1:
    with open(sys.argv[1]) as f:
        text = f.read()
else:
    text = sys.stdin.read()

matches = re.findall(pattern, text, re.DOTALL | re.MULTILINE)
if matches:
    last_match = matches[-1]
    print(last_match)
    if not re.search(p_end, last_match, re.DOTALL | re.MULTILINE):
        print(error_no_end)
else:
    print(error_no_match)
1

You could use an awk array with a toggle to buffer the latest block, and print the error text if the toggle is still set at the end (this is essentially an awk implementation of @anatoly_techtonik's python answer, I think):

awk '
  BEGIN{PROCINFO["sorted_in"]="@ind_num_asc"}

  /Transfer started/ {inblock=1; delete a;}
  /Transfer completed/ {a[FNR]=$0; inblock=0;}

  inblock == 1 {a[FNR]=$0}

  END {
    for (i in a) print a[i]; 
    if (inblock) 
      print "ERROR: transfer not complete by end of log file"
  }
' logfile

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.