I have a Python process which I run nightly via a crontab. It scrapes an API which I have limited access to so regularly has to stop and wait for an hour.
By my estimates the job should take 5-6 hours to complete.
At the moment I have a crontab as such:
05 01 * * * python3 /home/{my username}/path/to/process/main.py >> /var/log/process_name.log 2>&1
This is great and has proven to show that the file runs, but I can only check the output after it has finished running, which for me is not ideal. Considering the time it takes to run I would prefer cron to log output and errors in real time.
This is a bit more of a problem now as it seems my estimates are off and I can be waiting far longer than I think I should - in which case I would like to see what is being logged so I can understand why.
Is there a way to log cron output in real time?
import sys
followed bysys.stdout.flush()
in your python program.bash
script to runs someSQL
as well, at which point flushing throughpy
wouldn't work.