Kdump testing does not work on machines deployed by MAAS and JUJU

Question

I am trying to add kdump on my Openstack platform which is deployed by MAAS and JUJU.

I did several ways to do Kernel Crash Dump installation and testing. All the testing use Ubuntu OS 14.03 LTS version.

(1) Install kdump manually on host machines according to the ubuntu kernel-crash-dump guide. When I was going to issue the commands sudo sysctl -w kernel.sysrq=1 and echo c > /proc/sysrq-trigger with root permission, the console got stuck and showed few messages like the attached console image. I waited for a long time then reboot it, no crash log was written.

(2) By using JUJU charm: according to the steps in JUJU crash dump charm, I deployed ubuntu on host machine with juju deploy ubuntu and used JUJU GUI to deploy crash dump and add relation. This time it shows more details, but it gets stuck on CR2: 0000000000000000 like second attached image below.

From other Q&A info in google, the next step expected to do after stuck console should be

<blink>
[    0.000000] Initializing cgroup subsys cpuset 
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
...

(3) I only use MAAS to deploy Ubuntu OS on machine. And install kdump manually with the ubuntu kernel-crash-dump guide in (1). And the testing still gets stuck like "first" attached image.

Furthermore, I change password of Ubuntu account by executing sudo passwd ubuntu to do testing via Ubuntu account permission since it was created by MAAS (whoami shows Ubuntu account as root). But it shows the result of the "second" attached image.

(4) Install Ubuntu server OS and kernel-crash-dump manually on host machine. The testing was success and the crash log was generated on /var/crash as expected.

Each time the kdump config was checked before the testing by commands like example below and everything seems fine:

<blink>
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.13.0-77-generic root=UUID=8e8764a1-3d79-4f6e-945e-f30e42ea5f5a ro crashkernel=384M-:128M

# cat /sys/kernel/kexec_loaded
0
# cat /sys/kernel/kexec_crash_loaded
1

# kdump-config show
DUMP_MODE:        kdump
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 0x2c000000
current state:    ready to kdump

 kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-3.13.0-77-generic root=UUID=8e8764a1-3d79-4f6e-945e-f30e42ea5f5a ro irqpoll maxcpus=1 nousb" --initrd=/boot/initrd.img-3.13.0-77-generic /boot/vmlinuz-3.13.0-77-generic

I am still confused on why deployed Ubuntu OS by MAAS and JUJU can not execute kdump testing and have no idea to debug.

Thanks.

Answer 1

After reproduce this problem myself using JuJu/MAAS, I found out MAAS deployment install quite a few packages:

Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux

Some of the packages recreate the initrd file to a much larger size:

-rw-r--r-- 1 root root 24964912 Apr 18 16:32 /boot/initrd.img-3.13.0-85-generic
-rw-r--r-- 1 root root 24978648 Jun  7 09:32 /boot/initrd.img-3.13.0-87-generic

So the default crashkernel=384M-:128M parameter that added by kdump-tools became too small. After I changed the /etc/default/grub.d/kexec-tools.cfg

from

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=384M-:128M"

To

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=384M-:256M"

You need to do

update-grub

to update your grub configuration. After reboot, now you should be able to test your kernel crash and generate vmcore without problem.