Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a server which is monitored by Watchdog, and experiences reboots occasionally due to faulty network hardware I am unable to replace at the moment. As I read, Watchdog sends a SIGTERM to all processes, requesting a safe shutdown, and after a short time, sends a SIGKILL, which will stop the process immediately. However, in this case, it leads to data corruption since the main process of importance is not fully shut down yet and has unwritten data.

How long is this pause Watchdog takes between asking all processes to stop, and forcing them to stop? Is it hardwired within Watchdog, set in watchdog.conf(if it was, it never got documented in the manpage), or the same as another system setting? How may I change this setting?

Edit: I've found the timeout, but I am still looking for instructions on how to rebuild and integrate with the system properly.

share|improve this question

2 Answers 2

up vote 1 down vote accepted

I have posted some information about building, configuring and testing the Linux watchdog daemon here:

http://www.sat.dundee.ac.uk/~psc/watchdog/Linux-Watchdog.html

The short answer is you need to configure your system to build for this project:

sudo apt-get build-dep watchdog
sudo apt-get install build-essential automake libtool

Then get the source code, which you probably already have, but the latest can be done with these steps:

sudo apt-get install git
git clone git://git.code.sf.net/p/watchdog/code watchdog-code

Move to the code directory and prepare for compiling:

cd watchdog-code
autoreconf -i
./configure

Then to the source directory and compile it:

cd src
make clean
make

In the current directory you will have the new binaries. Test them before you make them "live" by using sudo make install, or at least make back-up copies of the system supplied programs. They are bloated compared to the system ones due to debug symbols, you can use the strip command if you want to reduce their size.

Can you say what sort of time you need for SIGTERM to work?

Edited to add:

If you are using the current GIT pull (14 Sep 2013) then edit shutdown.c and at line 363 change "safe_sleep(4);" to have your wanted timeout value in seconds. If editing the code for the system-supplied watchdog (as referenced above) take care not to sleep() for more than the hardware timeout (normally 60 seconds) as the system will simply reboot! That was the reason for the safe_sleep() function, to keep the watchdog fed while waiting.

share|improve this answer
    
I have a pretty strange process that needs up to a minute, which is obviously not a good time, but I can't fix that application. –  hexafraction Sep 13 '13 at 22:07
    
Making the delay user-configurable is, of course, possible so it might be worth posting that request to the watchdog project. –  Paul Crawford Sep 14 '13 at 8:06
1  
Just to add that I have added a user-configurable time to my experimental "V6.0" version of the watchdog daemon should you wish to try that. –  Paul Crawford Sep 19 '13 at 15:47

From the Watchdog source, shutdown.c, line 445, the pause is hardwired into Watchdog, and is five seconds.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.