I have a python program which, from time to time, crashes with a number of different exceptions usually due to network issues. The only way to get it to work is to restart it. How would I write a shell script to run the process, continually check if any of the output says Error and if so end the process and restart the command (let's call it ./command.sh)?

It would be fine for this to be in any other language (python/perl etc).

  • Why can't you just fix the python program? – geirha Jun 14 '11 at 18:06
  • Because python is not my forté – Callum Rogers Jun 14 '11 at 18:40
  • 1
    Also if it were a thrid party program, that's regularly updated, the next update might just bring back the problem... – con-f-use Jun 14 '11 at 18:43
  • That is also a problem, I'm pulling it from a git repo every couple of days and it would be annoying to add the change each time/make it work/fix the mergeconflicts – Callum Rogers Jun 14 '11 at 19:02

Since python exits with an return code != 0 when it throws an exception, you can just run this shell script:

while ! ./command.sh ; do : ; done

Of course if you just want to run the command all the time, just write:

while true ; do ./command.sh ; done

(this assumes ./command.sh doesn't fork and exit)

| improve this answer | |
  • We'll see if this works, now just to wait for an error :p – Callum Rogers Jun 14 '11 at 18:46

I did something similar to recover ted (Torrent Episode Downloader) from its regular crashes. Below is the script I used. Will explain it in further edits as I go.

Edit: How it works

Ted has a log file. Since it checks for new episodes every half an hour, there is a timestamp in that log, that's not older than half an hour. Of course the timestamp is not there if ted crashed in the mean time.

Edit: How that might help you

You could redirect your problematic program's output into a file. Write a script similar to mine, that checks this file. If it finds the bad string there, restarts the application (and removes the output file). You can ran the script in an endless loop or use cron. I chose to use cron so I have less process pollution and the checks don't have to be incredibly frequent anyway.

# indent-mode: spaces, indentsize: 4, encoding: utf-8
# © 2011 con-f-use@gmx.net.
# Use permitted under MIT license:
# http://www.opensource.org/licenses/mit-license.php (NO WARRANTY IMPLIED)

teddir='/home/confus/ted'   # Location of the ted executable
logpath='/home/confus/.Torrent Episode Downloader/log.txt' # Location of ted's log

# Get the process id for ted so it can be killed later
pid=$(lsof "$teddir/ted.jar" | grep -m1 java )
set -- $pid
echo "Ted is running unter pid: $pid."

# Check if the log is recent
search=$(date "+%b %d, %Y @ {0,1}%l:[0-9]{2} %p")
search=$(cat "$logpath" | grep -oE "$search")

# If ted is running and the log is not recent, kill crashed process and
#+restart it.
if [ -n "$pid" ] && [ ! -n "$search" ]; then
    echo "Killig $pid."
    kill "$pid" || exit 1
    echo "Restarting ted..."
    cd "$teddir"
    /usr/bin/java -jar ted.jar noTray

exit 0

In my case there was still an active process for ted after the crash, but ted wasn't doing anything. Kills it's process, then you can of course skip the pid-Parts.

P.S. It might or might not be a bad piece of code. There might be a much better way, but it works, and I put it together in a very short time.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.