2

Stumped by this one:

less, cat & grep are all able to list the content of an apache log file "/var/log/apache2/other_vhosts_access.log". But when I use awk to grab the last 5 minutes of the file then it occasionally works but most of the time it doesn't. No error given, so how to debug this one?

awk -v date="$(date --date='5 minutes ago' +"%d/%b/%Y:%T")" '$0~date {f=1}f' /var/log/apache2/other_vhosts_access.log >> /var/log/apache2/test/$(date +"%F_%T")_output.log

Does awk have issue with Apache writing to the file while trying to read it? Would it be better to pipe to awk from cat in this case?

My initial approach to grabbing the last 5 minutes was to use the timestamps in the log file plus some other things like IP address and byte count. All because multiple log entries will have the same timestamp. Doable but code intensive and hard to determine when logs are rotated.

The alternative is to count and preserve the last read line number (as an index) from the log file and at the next execution, read from there to the end of the file and updating the index. One would still have to record the timestamp of the last run to check that we're not reading more than 5 minutes at a time. Point is to graph data, so I'd rather lose data then deal with massive spikes which ruin graphs.

The above line is a lot more simple but not without it's own issues. How does one check if we're not accidentally duplicating data or missing data? It all hangs on cron's ability to execute at exact 300 second intervals. I was trying to test this but so far the above code doesn't yield results. dumped into an executable file and added to cron it hasn't worked yet. At least in my terminal it works occasionally.

5 * * * * /var/log/apache2/simple.sh 2>&1 > /dev/null

If I'm going about things completely wrong, please do let me know. I've found a few things regarding grabbing the last x minutes of log files and none seem to address my concerns listed.

  • 1
    awk doesn't have a problem, it's just another file ; there may be an error in the way you set up your date variable . . . Now, what exactly you're trying to do in that awk command ? print everything after specific date ? – Sergiy Kolodyazhnyy Nov 17 '15 at 18:08
  • @Serg exactly. Just figured out that if awk can't find the exact string I give it, it does what it's supposed to do and doesn't return anything. Doh!... If the date/time doesn't occur in the log file, the awk command doesn't match anything and returns nothing. I will have to read the timestamp of each line, convert it to epoch and compare to see if it falls in the range I'm interested in. – dmgeurts Nov 17 '15 at 18:22
  • 1
    I'd suggest you match part of the date, without seconds, match hours and minutes – Sergiy Kolodyazhnyy Nov 17 '15 at 18:42
  • That could work, on the presumption that there isn't a single minute in the day when apache doesn't log anything. Hadn't considered that so thank you. – dmgeurts Nov 17 '15 at 19:09
2

Just figured out why this command doesn't work... If the date/time doesn't occur in the log file, the awk command doesn't match anything and returns nothing... I will have to read the timestamp of each line, convert it to epoch and compare if it falls in the range I'm interested in.

Or as @Serg suggested, reduce the string to match by dropping the seconds. This does not guarantee a match, but does make it more likely. (ymmv)

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.