Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

One of my servers crashes once in a few days.

It does low traffic static web serving + low trafic dynamic web serving (PHP, local MYSQL with small data, APC, MEMCACHE) + some background jobs like XML file processing.

The only clue I have is that a few hours before the server dies it starts swapping (see screenshot http://awesomescreenshot.com/075xmd24 ) The server has a lot of free memory.

Server details: Ubuntu 11.10 oneiric i386 scalarizr (0.7.185) python 2.7.2, chef 0.10.8, mysql 5.1.58, apache 2.2.20, php 5.3.6, memcached 1.4.7 Amazon EC2 (us-west-1)

How can I detect the reason for the server crashes ? When it crashes its no longer accessible from the outside world.

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

Install linux-crashdump,at the very least, you'll get the kernel logging buffers just before the crash and all the data necessary for a professional to comb through the problem to determine the root cause.

share|improve this answer
    
When the server crashes, there is no access to it. How can I access the dump in this case? –  Nir Mar 13 '12 at 13:35
    
It writes it to disk so it can be retrieved once the system reboots. It literally boots a new linux kernel when the kernel panics, saves all the state to disk, and reboots. –  ppetraki Mar 13 '12 at 15:30
    
Hi ppetraki, Since this is a non persistent AWS EC2 instance, the disk goes away when the server dies. The instance doesn't boot but get terminated and its disk gets deleted. –  Nir Mar 18 '12 at 15:12
    
You don't seem very familiar with crashdump as a technology, this isn't new and has existed for ages. Here is an article regarding it's most recent re-implementation. lwn.net/Articles/108595 . There's nothing stopping you from changing the init script to netcat/scp the core dump to a central location for further post analysis. If you find this too challenging then seek commercial support. I can tell you as a kernel engineer, the first thing I'm going to ask for in a situation such as yours is "setup kdump and retrieve a core". –  ppetraki Mar 19 '12 at 14:46
    
You can use service like SeaLion in which you can see output of commands, executed on your server, online. So, even if your machine crashes, you will be able to see output of commands executed which will help you zeroing down on the problem. –  shubhansh Nov 11 '13 at 4:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.