grep form bottom and display lines after match

Question

How to achieve this task , I have pulled pork updating daily new signatures to the sid_changes.log and here is a search keyword "New Rules", I wanted to do bottom search for sid_changes.log and split those line after match "New Rules" to new file, I have came accross grep -A ,-B but these flags expect line numbers and I cannot determine line number as it will be dynamic....so these flags are not useful for me.

I have tried so far with diff command diff --changed-group-format="%>" --unchanged-group-format="" data1 data2 and Is there a better technique because I have to set cron jobs to first copy the sid_changes.log before it got updated and then diff it with new sid_changes.log, so this looks messed up solution ...I am expecting some better solution or design for this.

sid_changes.log
-=Begin Changes Logged for Wed Jan  2 19:05:09 2018 GMT=-

New Rules
        Reported CnC Server Port 40669 Group 1 (1:2405179)
        Server Port 45351 Group 1 (1:2405180)

-=Begin Changes Logged for Wed Jan  3 19:05:09 2018 GMT=-

New Rules
        ET CNC Shadowserver Reported CnC Server Port 40669 Group 1 (1:2405179)
        ET CNC Shadowserver Reported CnC Server Port 45351 Group 1 (1:2405180)
        ET CNC Shadowserver Reported CnC Server Port 47221 Group 1 (1:2405181)

-=Begin Changes Logged for Thu Jan  4 19:05:01 2018 GMT=-

New Rules
        BROWSER-OTHER Apple Safari javascript mutlibyte character escaping denial of service attempt (1:45354)
        BROWSER-OTHER Apple Safari javascript mutlibyte character escaping denial of service attempt (1:45355)
        BROWSER-OTHER Multiple browser long unicode string denial of service attempt (1:45302)
        BROWSER-OTHER Multiple browser long unicode string denial of service attempt (1:45303)
        BROWSER-PLUGINS UCanCode ActiveX clsid access attempt (1:45270)



ouputfile.log
-=Begin Changes Logged for Thu Jan  4 19:05:01 2018 GMT=-

New Rules
        BROWSER-OTHER Apple Safari javascript mutlibyte character escaping denial of service attempt (1:45354)
        BROWSER-OTHER Apple Safari javascript mutlibyte character escaping denial of service attempt (1:45355)
        BROWSER-OTHER Multiple browser long unicode string denial of service attempt (1:45302)
        BROWSER-OTHER Multiple browser long unicode string denial of service attempt (1:45303)
        BROWSER-PLUGINS UCanCode ActiveX clsid access attempt (1:45270)

Answer 1

I would use a perl command like this:

perl -ne 'if($_=~/^-=Begin/){@last=@block; @block=(); push(@block,$_);}else{push(@block,$_)}; END{@last=@block; print @last}'

Explanation:

• perl -ne ' -> read input line wise
• if($_=~/^-=Begin/){@last=@block; @block=();push(@block,$_);} -> flush last block to array last and start new block if the line begins with -=Begin
• else{push(@block,$_)}; -> add current line to block
• END{@last=@block; print @last}' -> flush last block and print it

To read from sid_changes.log and write to outfile.log, you can run it like this:

perl -ne '...' < sid_changes.log > outfile.log

Answer 2

You can use grep to find the line number of the lasts log entry separator and then use tail to only print lines from thereon:

lineno="$(grep -ne '^-=' sid_changes.log | tail -n 1 | cut -d ':' -f 1)"
tail -n "+$lineno" sid_changes.log > outfile.log

or as a one-liner

tail -n "+$(grep -ne '^-=' sid_changes.log | tail -n 1 | cut -d ':' -f 1)" sid_changes.log > outfile.log

---

If the input is not a file but a pipe/FIFO and you can only traverse it once you can use small program that tracks the log entries but only returns the last one:

#!/usr/bin/env python3
import sys

separator_prefix = '-='
last_entry = []

for line in sys.stdin:
    if line.startswith(separator_prefix):
        last_entry.clear()
    last_entry.append(line)

print(*last_entry, sep='', end='')

Usage:

python3 last-chang-entry.py < sid_changes.log > outfile.log