37

Following on from this question, I have written a simple upstart service (/etc/init/pms.conf) for my headless Ubuntu Server 11.04 box as follows:

start on filesystem and net-device-up IFACE=eth0
stop on runlevel [016]
respawn

exec /home/administrator/pms-current/PMS.sh

I can start (or stop) this service at will from the command line:

service pms start

And I can see that it is indeed running.

However, when I first boot my machine the service does not start. If I SSH into the box and check the service status I get:

$ service pms status
pms stop/waiting

My question is why is this happening? Why isn't my service starting on boot?

UPDATE 1: unsure whether my service was being started and subsequently dying or just wasn't being start at all, I added the following to PMS.sh:

echo "STARTED" > $STARTLOG

This obviously just gives me something to look for. I tested this by starting the service myself and then checking start.log. I then deleted the start.log and rebooted. It wasn't there after the restart, so it seems as though upstart definitely isn't starting my service. I suppose it could be dying at an earlier point in the process, but that seems rather unlikely given the simplicity of it all.

UPDATE 2: I've just upgraded to 11.10 which includes an upstart upgrade, but this problem still occurs.

UPDATE 3: As requested, I've booted with --debug. The output of cat /var/log/syslog | grep init is too long to place in the question, but you view it here.

UPDATE 4: More logs, this time the upstart conf is included at the top. Run 1 and run 2.

11
  • Are you sure the script was not executed? The pms stop/waiting message means that the upstart job was run and that all the commands terminated normally.
    – arrange
    Sep 24, 2011 at 15:17
  • Add the output of cat /var/log/syslog | grep init after enabling boot loggging for upstart using the instructions at Upstart Debugging Oct 26, 2011 at 2:20
  • @Anarci: please see update 3 in my question. Oct 26, 2011 at 12:33
  • Most users won't give out there email address like that, rather provide a link to a pastebin site like Ubuntu pastebin Oct 26, 2011 at 12:56
  • @Anarci: done - please see my question. Oct 26, 2011 at 13:00

12 Answers 12

20
+100

I would recommend increasing the verbosity of the job, e.g. by using pre-start/post-start entries.

pre-start script
  logger "pre-start for myprog"
end script

post-start script
  logger "post-start for myprog"
end script

# and for PMS itself:
script
  logger "just before executing PMS"
  exec /home/administrator/pms-current/PMS.sh
end script

Further information at http://upstart.ubuntu.com/cookbook/

Also have a look at http://upstart.ubuntu.com/wiki/Debugging

3
  • This really is doing my head in. I've tried a dozen different things off the back of your post. All have failed with varying obscure messages in the logs. My latest attempt resulted in init: pms main process (1329) terminated with status 143, which means naught to me. I can see that PMS.sh isn't even being started because the first thing it does is write to its own log, and that log entry isn't present. I can see my pre-start output, which tells me that the target file exists and is executable. I will pick this up again tomorrow, but if you have any ideas I'd love to hear them. Thanks. Oct 24, 2011 at 18:50
  • hi @KentBoogaart, I seem to have your same problem. Have you found a solution?
    – Daniele B
    May 22, 2014 at 19:50
  • @KentBoogaart I have the same problem as you? Any luck with yours ?
    – Mevin Babu
    Oct 13, 2014 at 8:53
14

What's probably happening here is that pms is starting before your network adapters come up, and probably before even the loopback adapter (lo). Assuming we're talking about PS3 Media Server, it's a networked service and it probably doesn't like starting up with no interfaces available.

Try changing your start on criteria to:

start on filesystem and net-device-up IFACE!=lo

Meaning, start after any "real" network interface is up. However, that might not be ideal, if eth0 is the next interface up, PMS starts, but you really want PMS to use wlan0, that won't do. The service will start but it might not have been able to pick the interface you wanted it to listen on. Assuming you know the interface you're going to stream over and it won't be changing, I would hardcode it into the job, e.g.:

start on filesystem and net-device-up IFACE=wlan0

On Oneiric (11.10), you can use the event static-network-up to wait for all statically configured devices. Which is nice because it allows you to write network-dependent jobs without hardcoding an interface. [Note: by "all statically configured devices", I'm referring to using /etc/network/interfaces instead of NetworkManager. It does not mean static in the sense of static IP vs. DHCP.]

9
  • This sounded like the trick, but it didn't work. I only have lo and eth0 but I used your second suggestion: start on filesystem and net-device-up IFACE=eth0. Still no go after a reboot. I've just noticed something in the PMS log that may be a lead. I'll investigate and get back... Sep 24, 2011 at 17:29
  • It's interesting. One thing I didn't mention is that I tried your original script and it worked on boot for my machine. I attributed that to just the luck of the draw (i.e. in my race condition, the good car won, and in yours the bad car won). I really can't see what other dependency we're missing here. Weird. Sep 24, 2011 at 17:49
  • 2
    Since you can start it after booting up, we must be missing another service dependency. One dirty hack that might work (but won't illuminate us at all) is to just drop in a sleep 10 -- or higher -- in a "pre-start script" before exec'ing the shell script. Sep 24, 2011 at 17:51
  • Sorry Mark - but we're on the same page. I tried the sleep 10 thing already in a pre-start script. No go. Then I tried deleting the debug.log altogether and rebooting. After boot I had the same service status, and no debug.log file, so I'm not convinced PMS is actually run at all. Is there an easy way to diagnose this? If I alter the PMS.sh to spit out some output, where will it go? I suppose I could always direct it to my own file - might give that a shot next. Sep 24, 2011 at 17:57
  • I just updated my question with more info. Sep 24, 2011 at 18:13
3

From examining your syslog the pms process starts with no errors but then after a short while its goal is changed from start to stop meaning it is killed.

This is slightly strange because you have added the repsawn clause so it should attempt to start again after it is stopped but it never does. So I'm guessing you removed the respawn clause.

Between the pms service starting and stopping only 2 services are started ufw and network-interface (eth0), and 1 is started udev-fallback-graphics.

It seems that you process pms is being started in parallel. Unfortunately the upstart documentation is a little bit hazy on the exact differences between start on ... vanilla and start on starting ... and start on started ....

Try changing your startup stanza to

start on started networking

or just too

start on net-device-up IFACE=eth0

The log output is slightly strange as the net-device-up event comes much later but pms starts before it.

This should ensure that your process only starts once all networking set up is finished i.e. the job has not only started but finished.

Also do not trust log output completely, early in the boot process logging output to any file does not always work. See the answer in Debugging Upstart

3

Managed to fix similar problem by using start on runlevel instead:

start on runlevel [2345]
3

I had the same problem and eventually I solved it simply with:

start on runlevel [2345]

without any net-device-up or started networking stuff

This is the complete upstart script, and it works perfectly:

# MyApp

description     "MyApp"
author          "me"

start on runlevel [2345]
stop on runlevel [016]

respawn

exec /usr/bin/myapp 2>> /var/logs/myapp.log
1

I came across chkconfig during my RHCSA/CE training:

sudo apt-get install chkconfig
sudo chkconfig pms on

You can check it's Oneiric man page for more details on it's capabilities.

1

I've found a solution for this but I don't understand it. If I move PMS out of /home/administrator and into /bin/pms with root as the owner, it all works fine.

If I leave it under /home/administrator/ but make sure root is the owner of everything bar the /home/administrator/ directory itself, it still doesn't work.

If I set administrator as owner of everything and change the pertinent part of my script to:

sudo su administrator -c '/home/administrator/pms-current/PMS.sh'

It still doesn't work.

I suppose for now I'll make a /home/root/ directory and move everything there, though I'd really like to fully understand this.

8
  • So the chkconfig didn't work either? Did you try giving the directory PMS.sh was in to root? If only your solution works, then go to the Upstart's Launchpad page and contact the devs directly.
    – Oxwivi
    Oct 26, 2011 at 17:08
  • And if it's enough to just move the .sh then just leave everything there and edit the script to point to that directory (or maybe even change directory?).
    – Oxwivi
    Oct 26, 2011 at 17:10
  • Yes, I tried making the whole PMS directory owned by root. Presumably it didn't work because /home/administrator/ is not owned by root. Oct 26, 2011 at 17:18
  • This doesn't make sense anyway, I regularly run scripts in my /home directory via upstart without problems, strange.
    – arrange
    Oct 26, 2011 at 17:26
  • Even stranger: I just tried everything under /home/root/ which is obviously owned by root. Didn't work. I moved everything back under /bin/pms and it worked again. So it seems trying to upstart PMS from under /home isn't working on my system. Oct 26, 2011 at 17:49
1

I had a similar "no start" problem when I have realized that my script depended on a file that was in my home, and the home was not accessible because was crypted with the standard ubuntu mechanism ( .Private).

start on local-filesystems event is (probably) emitted before the decryption process is ended.

1

Is your home directory on NFS? Sometimes root can't access NFS.

For the record, in my little test just now on 12.04:

  • start on started networking and start on network-interface-up INTERFACE=eth0 don't work, but

  • start on started network-interface INTERFACE=eth0 does.

Thanks to http://os4.org/wiki/upstart.html for pointing out that initctl list always shows job networking as stopped.

1
  • The link referenced is broken.
    – slm
    Apr 6, 2017 at 1:34
0

In my case, upstart service depended on the script located in vagrant synced folder. Solved the problem using the following line:

start on vagrant-mounted

More info: http://razius.com/articles/launching-services-after-vagrant-mount/

0

Similar to @xuhcc, I came here to find out why my Vagrant Upstart script wasn't running. The following is supposed to work:

start on vagrant-mounted

But doesn't in some builds due to the following bug.

https://github.com/mitchellh/vagrant/issues/6074

The workaround listed in the report worked great for me:

$ cat /etc/init/workaround-vagrant-bug-6074.conf 
# workaround for https://github.com/mitchellh/vagrant/issues/6074
start on filesystem
task

env MOUNTPOINT=/vagrant

script
  until mountpoint -q $MOUNTPOINT; do sleep 1; done
  /sbin/initctl emit --no-wait vagrant-mounted MOUNTPOINT=$MOUNTPOINT
end script

Worked great for me

0

it worked for me (i need start service after iface up):

start on started networking and net-device-up IFACE=wlan1 
stop on shutdown

respawn
respawn limit 10 10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .