7

We've had problems with tripping a breaker and I think we just did it again, but I need to confirm that 1 of our boxes rebooted due to a power outage and not for another mundane reason.

I have gleaned the following information from the logs, but wonder if there's something more telling:

Approx time of restart:
May 19 19:54:57 atlas07 

$ last
davidpar pts/0        10.1.2.12        Sun May 19 19:54   still logged in
reboot   system boot  3.5.0-30-generic Sun May 19 19:44 - 20:19  (00:34)
davidpar pts/0        10.1.2.83        Tue May 14 04:14 - 04:21  (00:06)
...


$ tail -n 500 syslog
<cut>
May 19 19:44:58 atlas07 ntpdate[1489]: adjust time server 91.189.94.4 offset 0.000047 sec
May 19 19:45:04 atlas07 CRON[1492]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May 19 19:54:57 atlas07 dbus[1012]: [system] Activating service name='org.freedesktop.ConsoleKit' (using servicehelper)
May 19 19:54:57 atlas07 dbus[1012]: [system] Activating service name='org.freedesktop.PolicyKit1' (using servicehelper)
May 19 19:54:57 atlas07 polkitd[1565]: started daemon version 0.104 using authority implementation `local' version `0.104'
<cut>
0

3 Answers 3

6

The last command will display crash for the end time of a session if the shutdown was not clean, which would include a power outage. Example entry:

myuser pts/1        myhostname  Wed Sep 28 14:18 - crash  (00:23)

Looking at your last entries (assuming this is all of them) there are no crashes listed there, so no power outage occurred while this system was on. The reboot that is listed here looks to be a commanded reboot and not a crash. If you didn't manually reboot this box, you then a program with root access did.

3
  • Is there a way to figure out what process rebooted then? We lost all nodes in a single 4-node server simultaneously, it sure smells of power outage (and we've had power issues recently as well). May 20, 2013 at 3:58
  • 1
    L- 4 node server without UPS? Suicide... try to manage at least some power backup.
    – Dee
    May 20, 2013 at 5:29
  • This is running in a datacenter, the datacenter has backup systems in place, the problem is how our power is balanced (we need to fix it certainly). But for now I need to be 100% certain that this was a power outage so I can pull the fire alarm. May 20, 2013 at 5:53
3

Your boxes are not connected to UPS? If you do so, usually there is a logging and support for this and you will clearly see when power is going down on UPS and when server is going down due to ups secure shutdown. Maybe you will be interrested into this question: https://unix.stackexchange.com/questions/13714/how-to-properly-prepare-a-server-for-power-outages7

When there is not any UPS it is hard to "write a log" when there is no power. This is impossible. You can find power outage indirectly to cut out other reasons.

I recommend to monitor all important parameters of your boxes to decide, what is shutting down your system. Can be a component failure as mainboard or memory (usually some bluescreens/mem michmatched), or CPU and fans (overheating ends up in shutdown), or HDD r/w errors (bad clusters on system resources), each has some typical symptoms.

Look into /var/log/syslog /var/log/kern.log if there is something interesting.

Power outage can be recorded in system BIOS. You can try to read system BIOS log, if there is any, but it would be elaboration as each server is different in that manner. But this will not necessary tell you what is going on and what is the reason of "server switches off". Try dmidecode command to get BIOS details, but I doubt there will be a log.

or.. if you don't want to invest into UPS, you can at least monitor the power line via on-site notebook (has baterry and can read and log the power manager status) try to log a grep 'Adapter' from acpi -V, as it should write something like Adapter 0: on/off-line

acpi -V |grep 'Adapter'
1
  • No UPS sadly, I expect that we've tripped a surge protector (something we encountered 2 weeks ago as well). May 20, 2013 at 3:59
1

Here is a simple bash script that will tell you if the last shutdown was a safe or unsafe restart.

if [[ "$(last -x | tac | grep -B1 boot | tail -2 | grep shutdown)" == "" ]]; then
  echo "['$(date)'] Last shutdown: Uncontrolled (not safe). Possible power failure."
else
  echo "['$(date)'] Last shutdown: Controlled (safe)"
fi

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .