2

I have a Python process which I run nightly via a crontab. It scrapes an API which I have limited access to so regularly has to stop and wait for an hour.

By my estimates the job should take 5-6 hours to complete.

At the moment I have a crontab as such:

05 01 * * * python3 /home/{my username}/path/to/process/main.py >> /var/log/process_name.log 2>&1

This is great and has proven to show that the file runs, but I can only check the output after it has finished running, which for me is not ideal. Considering the time it takes to run I would prefer cron to log output and errors in real time.

This is a bit more of a problem now as it seems my estimates are off and I can be waiting far longer than I think I should - in which case I would like to see what is being logged so I can understand why.

Is there a way to log cron output in real time?

  • 1
    Have you tried to flush the output? import sys followed by sys.stdout.flush() in your python program. – Jos Apr 17 '15 at 7:33
  • @jos good point and I could potentially do that, although Ideally I would eventually move this into a bash script to runs some SQL as well, at which point flushing through py wouldn't work. – Scironic Apr 17 '15 at 7:36
  • 1
    Shell scripts flush on a line by line basis. Does that help? – Jos Apr 17 '15 at 7:46
  • Allow me to convert this to an answer. – Jos Apr 17 '15 at 7:52
1

What you need is a mechanism called flushing: forcing a process to write its output, which would otherwise remain buffered until completion.

Shell scripts flush on a line by line basis. If you need the output of a shell script before the script is ended, make sure it contains a newline.

Python programs can explicitly flush their stdout by calling the flush method:

import sys
sys.stdout.flush()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.