4

I'm running my application on a VPS under Ubuntu 12.04 LTS.

And I have a problem with one daemon process for one of my applications. This process is called "delayed_job" (If not familiar - this is just a background task handler).

The problem with this process is that sometimes it gets killed (I suppose it is because I have limited amount of RAM on my server - only 1GB).

But the main problem with this thing is that it is not capable of booting again after a crash or "kill" signal, unlike "unicorn" (it is a Rails app server) processes, which are always getting re-booted no matter what happens to them.

And yes, this IS a very big deal, because a lot of application features are using background tasks.

Same story happens when there are maintenance works on the VPS (initiated by the owner of the VPS) and after the restart the "delayed_job" process is not present in the system anymore.

This is the command I run every single time to boot it up: RAILS_ENV=production script/delayed_job start

It is a well-known problem, but the only solution I found on the Internet is this article: http://www.alexreisner.com/code/upstart that suggests using Linux's "upstart" feature with the "respawn" option that is capable of re-booting the process if it gets killed or crashed.

Even though the article is a bit out-dated, I found out that Ubuntu 12.04 should support this feature and I've created a symlink in /etc/init directory (I've called it: delayed_job.conf) to the "delayed_job" file that I'd put in one of my applications folders (app_name/config, to be exact) - I've done everything this article says.

My problem is: when I try booting this new process (start delayed_job) in the console I get:

delayed_job start/running, process 6000

But in fact - no "delayed_job" process is getting created at all.

And the status (status delayed_job) of the initctl process is still: delayed_job stop/waiting

After I execute kill -9 6000 I get -bash: kill: (6000) - No such process

So, this means that nothing gets executed. I've tried running this numerous times under different conditions - nothing, no avail. Is simply does not work.

Is there any thing I could try to make it work, or it is just useless ?

4

Sounds like your EXPECT stanza is incorrect or not there. Upstart is tracking the wrong pid. See the cookbook for instructions on how to use EXPECT. Note the warning about how important that section is to understand.

Note that if your application is too forky (meaning it forks more than twice) for Upstart to track its pid, you may be able to track it yourself. The app may have a mechanism to write a pid file, or you may be able to capture it with start-stop-daemon. See this answer for an example, specifically the pg_agent.conf file.

Edit:

Note, if Upstart can't track your app's pid, you won't be able to use the RESPAWN stanza. In this case Upstart may not suit your needs. Perhaps a competing product like forever will. Don't know.

  • Yeah. I did not use "expect" stanza at all. Thanks! I'll take a look! – Dmitri Feb 26 '14 at 13:43
  • Well, now the "start" and "stop" both work, but, the 'delayed_job' process is not booting-up (according to ps aux | grep "delayed_job"). I tried both options: fork and daemon - neither works. – Dmitri Feb 26 '14 at 21:51
  • What is the fork count? – Brian.D.Myers Feb 26 '14 at 22:56
  • should I count it using strace ? What should I do if it is greater than 2 (deamon) ? – Dmitri Feb 27 '14 at 8:59
  • Yes, count with strace. See the last sentence of my edited answer for ideas if it's > 2. – Brian.D.Myers Feb 27 '14 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.