How to capture lines between two strings from a file, but only the last occurrence?

Question

I have a log file which is output by a script, the log file is rotated daily. It will contain the strings

Transfer started at timestamp 

and

Transfer completed successfully at timestamp

repeatedly, as the mentioned transfer will take place hourly. The timestamps will have been previously created with date.

• I want to capture the last instance of these two strings, and everything in between, into a separate file.
• If the started string is found near the end of the log file, with no following completed string, I want to capture everything up to EOF and output an error message to say that the end string was not found.

I'm guessing I'll need to use sed or awk but am really inexperienced with them. I want to use the command in a bash script, and understand what each part is doing, so some explanation would be very useful.

An example chunk of log file:

ERROR - Second tech sync failed with rsync error code 255 at Fri May 27 13:50:4$
--------------------------------------------------------------------
After_sync script completed successfully with no errors.
Main script finished at Fri May 27 13:50:43 BST 2016 with PID of 18808.
--------------------------------------------------------------------
Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.
37 approvals pending.
Transfer completed successfully at Fri May 27 14:05:16 BST 2016
--------------------------------------------------------------------
Local repository verification started at Fri May 27 14:35:02 BST 2016
...

The desired output:

Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.
37 approvals pending.
Transfer completed successfully at Fri May 27 14:05:16 BST 2016

However, if the log file was like this:

ERROR - Second tech sync failed with rsync error code 255 at Fri May 27 13:50:4$
--------------------------------------------------------------------
After_sync script completed successfully with no errors.
Main script finished at Fri May 27 13:50:43 BST 2016 with PID of 18808.
--------------------------------------------------------------------
Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.

I would want to output:

Transfer started at Fri May 27 13:50:45 BST 2016
Logs transferred successfully.
Images transferred successfully.
Hashes transferred successfully.
ERROR: transfer not complete by end of log file

Answer 1

When I hear "I want to do X with the last something in the file", I think:

• reverse the file
• do X with the first something in the file
• reverse the output of X

In code:

tac logfile | awk '
    BEGIN {text = "ERROR: transfer not complete by end of log file"}
    /^Transfer completed successfully/ {text = ""}
    {text = text ORS $0}
    /^Transfer started at / {print text; exit}
' | tac

Since we are reading the log file from the bottom up, I start with assuming the transfer is not completed. If I see the "transfer completed" message, we can throw out whatever we've captured so far. We save each line. When we see the "transfer started" line, we know we have seen all of the last transfer in the file: print out the (reversed) captured text and exit awk.

Answer 2

Just use Python. I don't really have time, but I'd start with this:

#!/usr/bin/env python

start = "Transfer started at"
end = "Transfer completed successfully"
buffer = ""
log = False

for line in open('logfile.log'):
  if line.startswith(start):
    buffer = line
    log = True
  elif line.startswith(end):
    buffer += line
    log = False
  elif log:
    buffer += line

open('output.log', 'w').write(buffer)

if log == True:
  print("End string was not found")

Answer 3

Python all the things, but let regular expressions do the work for you!

Paste the script below to any file, e.g. logfilter.py and make it executable using the command chmod +x logfilter.py.

Then you may run it like this, assuming it's located in the current directory:

./logfilter.py logfile.txt

This will make it process the file logfile.txt.

However, if you do not pass it any command-line arguments, it will wait for data on standard input. This means you can also pipe data into it. The following example processes data from the clipboard (needs xsel installed to access the clipboard):

xsel -ob | ./logfilter.py

---

The script:

#! /usr/bin/env python3

p_start = r'^Transfer started at .*?$'
p_end   = r'^Transfer completed successfully at .*?$'

error_no_match = 'ERROR: no match found'
error_no_end   = 'ERROR: transfer not complete by end of log file'

pattern = r'{p0}(?!.*{p0})(?:.*?{p1}|.*)'.format(p0=p_start, p1=p_end)

import sys, re
if len(sys.argv) > 1:
    with open(sys.argv[1]) as f:
        text = f.read()
else:
    text = sys.stdin.read()

matches = re.findall(pattern, text, re.DOTALL | re.MULTILINE)
if matches:
    last_match = matches[-1]
    print(last_match)
    if not re.search(p_end, last_match, re.DOTALL | re.MULTILINE):
        print(error_no_end)
else:
    print(error_no_match)

Answer 4

You could use an awk array with a toggle to buffer the latest block, and print the error text if the toggle is still set at the end (this is essentially an awk implementation of @anatoly_techtonik's python answer, I think):

awk '
  BEGIN{PROCINFO["sorted_in"]="@ind_num_asc"}

  /Transfer started/ {inblock=1; delete a;}
  /Transfer completed/ {a[FNR]=$0; inblock=0;}

  inblock == 1 {a[FNR]=$0}

  END {
    for (i in a) print a[i]; 
    if (inblock) 
      print "ERROR: transfer not complete by end of log file"
  }
' logfile