restart my app server automatically through upstart script

Question

I have a Ubuntu 12.04 machine in which I am running my C++ app server which I am controlling through upstart script. Meaning whenever my app server crashes my upstart script will restart them back automatically and whenever my ubuntu machine gets rebooted/restarted my upstart script will restart my app server again.

Below is my upstart script:

start on runlevel [2345]
stop on runlevel [016]

chdir /opt/process/process_server
respawn

post-start script
    echo "App server started at `date +"%F %T"` on `hostname -f`" | mailx -r "abc@host.com" -s "Process Started" "pqr@host.com"
end script

post-stop script
  sleep 30
end script

limit core unlimited unlimited
limit nofile 8092 8092
setuid caprti
exec ./process_server --config_file=../config/process.init

Now weird thing is, I have see this atleast twice - My ubuntu machine was restarted and I don't know how and who did that but my app server was not restarted at all and I don't know why? I have tested this multiple times by myself rebooting that same box by typing "sudo reboot" and whenever that machine comes back up, I see my app server gets restarted automatically.

Under what condition during reboot of my Ubuntu box will not result in my app server getting restarted from my above upstart script? Is there anything I am missing in my Upstart script?

Also how do I verify that my Upstart script tried to restart my app server during the last reboot but it failed for some reason if that's what has happened with my for those two times?

Answer 1

I have some notes or hints that may help:

• respawn stanza without explicitly specifying limits has defaults.

  6.28 respawn

  Further note that if the job does not specify the respawn limit stanza as well as the respawn stanza, the job will have the default respawn limit applied (see respawn limit).

  6.29 respawn limit

  Respawning is subject to a limit. If the job is respawned more than COUNT times in INTERVAL seconds, it will be considered to be having deeper problems and will be stopped. Default COUNT is 10. Default INTERVAL is 5 seconds.

  Use:

  respawn 
  respawn limit unlimited
  

• Check its logs as in how to debug upstart scripts?. You may create your own custom logs to track more specific problems use pre-start, post-start & post-stop scripts and check variables like RESULT and PROCESS, see Upstart: report (email, log) on respawn and failure

• start on runlevel [2345]!!! Are you sure the runlevel is the only needed condition to start this service? examples: does that c++ daemon need filesystem, networking.. See What events are available for Upstart?

  Someone used start on stopped rc to avoid looking for each event, source: Start jackd with realtime priority with upstart

References:

• Upstart Intro, Cookbook and Best Practises: respawn , respawn limit

Answer 2

Debugging

By default system store logs of upstart files in /var/log/upstart. generally , it store with name_of_upstartfile.log in /var/log/upstart. Look into /var/log/upstart/file.log . where file.log is log of your init file /etc/init/file.conf. you can easy identify what you are missing in upstart script.

check syntax of upstart file init-checkconf /etc/init/file.conf or with more debugging mode init-checkconf -d /etc/init/file.conf

use start on local-filesystems or start on (local-filesystems and net-device-up IFACE!=lo) instead of start on runlevel [2345]. you are executing command that need local-filesystem and if networking also nedded than add networking or check other requirement. still problem then post log or error .