2

I have a Python process which I run nightly via a crontab. It scrapes an API which I have limited access to so regularly has to stop and wait for an hour.

By my estimates the job should take 5-6 hours to complete.

At the moment I have a crontab as such:

05 01 * * * python3 /home/{my username}/path/to/process/main.py >> /var/log/process_name.log 2>&1

This is great and has proven to show that the file runs, but I can only check the output after it has finished running, which for me is not ideal. Considering the time it takes to run I would prefer cron to log output and errors in real time.

This is a bit more of a problem now as it seems my estimates are off and I can be waiting far longer than I think I should - in which case I would like to see what is being logged so I can understand why.

Is there a way to log cron output in real time?

4
  • 1
    Have you tried to flush the output? import sys followed by sys.stdout.flush() in your python program.
    – Jos
    Apr 17, 2015 at 7:33
  • @jos good point and I could potentially do that, although Ideally I would eventually move this into a bash script to runs some SQL as well, at which point flushing through py wouldn't work.
    – Scironic
    Apr 17, 2015 at 7:36
  • 1
    Shell scripts flush on a line by line basis. Does that help?
    – Jos
    Apr 17, 2015 at 7:46
  • Allow me to convert this to an answer.
    – Jos
    Apr 17, 2015 at 7:52

1 Answer 1

1

What you need is a mechanism called flushing: forcing a process to write its output, which would otherwise remain buffered until completion.

Shell scripts flush on a line by line basis. If you need the output of a shell script before the script is ended, make sure it contains a newline.

Python programs can explicitly flush their stdout by calling the flush method:

import sys
sys.stdout.flush()

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .