1

I am trying to write a script to collect the last 24 hours logs from nginx log files. My script is collecting all the logs from the log file and I need only last 24 hours errors.

Script to collect last 24 Hours nginx access.log and error.log

awk -vDate=`date -d'now-24 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /var/log/nginx/access.log > /data/production_logs/nginxaccess.log
awk -vDate=`date -d'now-24 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /var/log/nginx/error.log > /data/production_logs/nginxerror.log

2nd Script:

egrep 'Error|error|Exception|failed|Unhandled|err|Err' /data/production_logs/myapp.log > /data/production_logs/myapp_error.log

Looking Script Example like below:

egrep 'Error|error|Exception|failed|Unhandled|err|Err' /var/log/nginx/error.log > /var/log/nginx/last24hourlogs.log

To grep error exception at above from last 24 hours logs only and save it in as last24hourlogs.log

Log Format required:

2016/11/27 13:55:00 [error] 6822#0: *14569 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 223.182.171.4, server: myappserver
2016/12/03 12:51:26 [error] 6820#0: *19094 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 157.48.95.8, server:

tail -f /home/example.com/.forever/bdapp.log

2016/12/19 12:30:51 [error] 2147#0: *5647 open() "/usr/share/nginx/html/example.com/myapp_email-templates/social-01.png" failed (2: No such file or directory), client: 66.249.84.191, server: example.com, request: "GET /myapp_email-templates/social-01.png HTTP/1.1", host: "example.com"
2016/12/19 12:30:51 [error] 2147#0: *5646 open() "/usr/share/nginx/html/example.com/myapp_email-templates/social-02.png" failed (2: No such file or directory), client: 66.249.84.128, server: example.com, request: "GET /myapp_email-templates/social-02.png HTTP/1.1", host: "example.com"
19
  • I don't have the file, what does a line exactly look like? Dec 14 '16 at 9:54
  • @Zanna Ah, great :) Dec 14 '16 at 9:56
  • @Zanna shouldn't be too difficult :) Dec 14 '16 at 10:05
  • Hi Ramesh, I have a perfectly working script, but I need to know the exact format of your lines, and also the script can be optimized if I knew if your file was sorted (no doubt) and how: latest messages first, or at the bottom of your file. Dec 14 '16 at 11:24
  • 1
    @RameshChand in order to help you, we need to see the execat file you are trying to parse. Don't show screenshots (we need to be ab;e to copy it for testing) and don't add comments. Please edit your question and add i) an example of the file you want to process (if there are multiple date formats, we need to see all of them) and ii) the output you expect from that example input.
    – terdon
    Dec 20 '16 at 10:32
2

Reading a log file over x (last) hours, looking for specific lines

Important notes

  • The answer below was written, based on the example OP provided, assuming the output example is an exact copy of how lines occur in the log file. This is essential for correct parsing out the date; if either the position or the format of the time stamp is different, it will fail!
  • Due to missing sorting information, the script could not be optimized for performance; all lines need to be checked, with the information we currently have.
  • Unclear also is if the log file reports in UTC or local time, and in what time the report should be produced. "last 24 hrs" possibly needs to be corrected by the local time difference.

The script

#!/usr/bin/env python3
import time
import calendar
import sys

#--- set conditions below 
matches = ['Error', 'error', 'Exception', 'failed', 'Unhandled', 'err', 'Err']
# ---

pattern = "%Y/%m/%d%H:%M:%S"

source = sys.argv[1]
report = sys.argv[2]
last_hrs = sys.argv[3]

# shift =  time.timezone
shift = 0
now = time.time()

def convert_toepoch(pattern, stamp):
    """
    function to convert readable format (any) into epocherror
    """
    return int(time.mktime(time.strptime(stamp, pattern)))

with open(source) as infile:
    with open(report, "wt") as outfile:
        for l in infile:
            try:
                # parse out the time stamp, convert to epoch
                stamp = "".join(l.split()[:2])
                tstamp = convert_toepoch(pattern, stamp)
                # set the conditions the line has to meet
                if now - tstamp - shift <= int(last_hrs)*3600:
                    if any([s in l for s in matches]):
                        outfile.write(l)
            except (IndexError, ValueError):
                pass

How to use

  1. Copy the script into an empty file, save it as get_log.py
  2. Run it with the sourcefile, the output file and the time as arguments:

    python3 /path/to/get_log.py <logfile> <ouput_file> 24
    

As mentioned, possibly the time (24) needs to be fixed by the local timezone. Please let me know.

What it does

  • The script looks for lines with a time stamp, showing a time within the defined time period (x-hours back from now), comparing time from epoch. In case of a match, it looks if any of the conditional strings is in the file.
  • If so, the line is written to the report

EDIT

OP mentioned it didn't work. A test on both examples OP posted on request however, shows the script does the job perfectly:

Why does it work?

  • Op's example- timestamp:

    2016/11/27 13:55:00
    

    is converted into the format:

    "%Y/%m/%d%H:%M:%S"
    

    by the line:

    stamp = "".join(l.split()[:2])
    

    and subsequently converted to epoch:

    tstamp = convert_toepoch(pattern, stamp)
    
  • The line:

    if now - tstamp - shift <= int(last_hrs)*3600:
    

    selects lines, stamped within last_hrs from now.

  • The line:

    if any([s in l for s in matches]):
    

    subsequently looks if any of the strings:

    ['Error', 'error', 'Exception', 'failed', 'Unhandled', 'err', 'Err']
    

    occurs in the line.

As mentioned, I tested it thoroughly with the exact examples OP provided, and cannot come to another conclusion than that the script does its job.

26
  • There is no ignore case in python? matches = ['Error', 'error', 'Exception', 'failed', 'Unhandled', 'err', 'Err'] Dec 14 '16 at 15:24
  • @KasiyA Not that I know, but in can be shorter though, if we say: if err in l.lower(). Thanks for the hint! Dec 14 '16 at 15:25
  • @KasiyA ^...But that would possibly clash with other options, which might be case specific. Dec 14 '16 at 15:28
  • Thanks for the script let me check it and share the update soon. Dec 15 '16 at 5:02
  • 1
    @RameshChand You must have copied/paste incorrectly. I just tried the posted code, and it does not have an indentation error. Please try again. Please note you that you should not change anything in indentation and/or spaces. They are essential in python. Dec 15 '16 at 15:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.