2

I'm running Ubuntu Server 14.04 on a machine that is experiencing random crashes under load. The server doesn't restart independently, but it becomes inaccessible via ssh or direct KVM connection.

I suspect CPU issues, but I'm looking for a smoking gun, so I'm following the instructions here:

https://wiki.ubuntu.com/Kernel/CrashdumpRecipe

However, no logfile appears in /var/crash after either a natural or induced crash.


Details:

(I induced a crash between each change described here, with no change.)

Everything went smoothly until I got to this step:

$ cat /sys/kernel/kexec_crash_loaded
0

Expected output was 1. A bit of digging led me to /etc/default/kdump-tools, where I set USE_KDUMP=1. When that didn't work, I added KDUMP_SYSCTL="kernel.panic=60 kernel.panic_on_oops=1" based on the sysctl documentation. Still no joy, so I modified the param directly with sysctl -w kernel.panic=60 to supposedly add extra time for kdump to do its thing.

In every case, I would run:

echo c | sudo tee /proc/sysrq-trigger

The computer would crash and reboot as expected, but all I'd see is this:

$ ls /var/crash
kexec_cmd

/var/log/kern.log contains only log entries from boot, not from the crashes. (Not sure if this is expected, but I thought I'd mention it anyway.)

Is there something wrong with my configuration?

  • Did you ever figure this out? I'm having the same problem. – Will Jan 5 '16 at 20:40
0

If /boot is on a separate partition, Kernel Crash Dump on Ubuntu will fail because of a ---> bug which still exists in 14.04 Trusty, can you believe it? At least it is the case for me using LVM and a separate /boot.

The workaround is to

  1. unmount /boot and then mount it to some other mount point, e.g. /mnt
  2. copy the contents over to /boot (the one on the same block device as /). e.g. rsync -axHAX --progress --stats /mnt/ /boot
  3. Trigger a crash by using echo c | sudo tee /proc/sysrq-trigger
  4. If all good, you'll see kernel crash dump in /var/crash in forms of (uname -r)-yyyymmddHHmm.crash and a yyyymmddHHmm directory with dmesg and dump.

If you want to analyze the crash dump, you'll need crash, run it like below:

crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/yyyymmddHHmm/dump.yyyymmddHHmm

For more info regarding crash, read the manual.

BTW: DO NOT forget to reload the kdump-tools configuration after changing /etc/default/kdump-tools by kdump-config load.

| improve this answer | |
  • will there be any issue if you unmount /boot? As the grub and the image are on this partition? Will the system reboot successfully after unmount /boot? – Robin May 31 '18 at 17:45
0

Maybe it is the reserved memory is too small.(my problem caused by this reason).

I think you should check the right thing from three steps. First, check your config is follow the [Ubuntu kernel Crash dump]:https://help.ubuntu.com/lts/serverguide/kernel-crash-dump.html

Second, dmesg|grep -i crash, to check the reserved memory is ok.

Third, service kdump-tools status to check the load kdump kernel is ok.

In the third step, the log is very important, your can see /var/log/syslog to find the logs. and then identify the reason.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.