1

I'm trying to get kernel crashdump to work properly in Ubuntu 15.10 with the stock 4.2.0-22-generic kernel. I have followed the methods described here and here precisely. But, when I manually trigger a crash via:

echo c | sudo tee /proc/sysrq-trigger

the system crashes, and reboots, but no crash output is saved in /var/crash.

As this is EC2, I don't have a read/write console--I can only get read-only console output, and I'm not seeing much useful output:

[  473.666303] sysrq: SysRq : Trigger a crash
[  473.668278] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  473.671624] IP: [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[  473.672244] PGD 3e235c067 PUD 3e2351067 PMD 0
[  473.672244] Oops: 0002 [#1] SMP
[  473.672244] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev xen_fbfront intel_rapl fb_sys_fops iosf_mbi input_leds serio_raw parport_pc 8250_fintek i2c_piix4 parport mac_hid autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea aesni_intel aes_x86_64 sysfillrect lrw sysimgblt gf128mul ttm glue_helper ablk_helper drm_kms_helper cryptd psmouse drm ixgbevf pata_acpi floppy
[  473.672244] CPU: 3 PID: 2814 Comm: bash Not tainted 4.2.0-22-generic #27-Ubuntu
[  473.672244] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
[  473.672244] task: ffff8803d1d86e00 ti: ffff8803dc414000 task.ti: ffff8803dc414000
[  473.672244] RIP: 0010:[<ffffffff814c79e6>]  [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[  473.672244] RSP: 0018:ffff8803dc417e28  EFLAGS: 00010246
[  473.672244] RAX: 000000000000000f RBX: 0000000000000063 RCX: 0000000000000000
[  473.672244] RDX: 0000000000000000 RSI: ffff8803ff2ce938 RDI: 0000000000000063
[  473.672244] RBP: ffff8803dc417e28 R08: 0000000000000002 R09: 000000000000024d
[  473.672244] R10: 000000000000a614 R11: 000000000000024d R12: 0000000000000004
[  473.672244] R13: 0000000000000000 R14: ffffffff81cb48e0 R15: 0000000000000000
[  473.672244] FS:  00007fecdb3ca700(0000) GS:ffff8803ff2c0000(0000) knlGS:0000000000000000
[  473.672244] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  473.672244] CR2: 0000000000000000 CR3: 00000003e234b000 CR4: 00000000001406e0
[  473.672244] Stack:
[  473.672244]  ffff8803dc417e58 ffffffff814c821a 0000000000000002 fffffffffffffffb
[  473.672244]  ffff8803dc417f18 0000000000000002 ffff8803dc417e78 ffffffff814c86a3
[  473.672244]  0000000000000002 ffff8803f816c900 ffff8803dc417e98 ffffffff81266aa2
[  473.672244] Call Trace:
[  473.672244]  [<ffffffff814c821a>] __handle_sysrq+0xea/0x140
[  473.672244]  [<ffffffff814c86a3>] write_sysrq_trigger+0x33/0x40
[  473.672244]  [<ffffffff81266aa2>] proc_reg_write+0x42/0x70
[  473.672244]  [<ffffffff811fca68>] __vfs_write+0x18/0x40
[  473.672244]  [<ffffffff811fd3f6>] vfs_write+0xa6/0x1a0
[  473.672244]  [<ffffffff810c3e21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[  473.672244]  [<ffffffff811fe0e5>] SyS_write+0x55/0xc0
[  473.672244]  [<ffffffff8121b31f>] ? __close_fd+0x8f/0xb0
[  473.672244]  [<ffffffff817f02b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[  473.672244] Code: 45 3b 7d 34 75 e5 4c 89 ef e8 f7 f7 ff ff eb db 0f 1f 44 00 00 0f 1f 44 00 00 55 c7 05 a8 74 a2 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 48 89 e5 53 8d
[  473.672244] RIP  [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[  473.672244]  RSP <ffff8803dc417e28>
[  473.672244] CR2: 0000000000000000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.2.0-22-generic (buildd@lcy01-22) (gcc version 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) ) #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 (Ubuntu 4.2.0-22.27-generic 4.2.6)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 crashkernel=384M

...
[    3.021894] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
...
[  OK  ] Started memcached daemon.
         Starting LSB: Execute the kexec -e command to reboot system...
...
[  OK  ] Started LSB: Record successful boot for GRUB.
[  OK  ] Started LSB: automatic crash report generation.
[  OK  ] Started LSB: Execute the kexec -e command to reboot system.
...
[  OK  ] Started LSB: Load kernel image with kexec.
ondemand.service
rc-local.service
grub-common.service
         Stopping LSB: Start NTP daemon...
apport.service
kexec.service
...
lxc.service
[  OK  ] Started LXC Container Initialization and Autoboot Code.
         Starting Container hypervisor based on LXC - boot time check...
[   34.181647] kdump-tools[773]: Starting kdump-tools:  * loaded kdump kernel
kdump-tools.service
[  OK  ] Started Kernel crash dump capture service.
[  OK  ] Started Container hypervisor based on LXC - boot time check.

And then the system is fully back online, with nothing in /var/crash except .lock and kexec_cmd.

I've tried crashkernel=128M, crashkernel=256M, crashkernel=384M, 512M, 256@0, 256@16M, etc.

I even tried enabling SSH in /etc/default/grub.d/kexec-tools.cfg, with a machine I've verified I can access from this machine, with a manual SSH_KEY configured that exists, works, and has proper permissions set, but the remote machine doesn't show a connection attempt at all.

The output of kdump-config show looks proper:

DUMP_MODE:        kdump
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 0x2c000000
SSH:              ubuntu@10.xxx.xxx.xxx
SSH_KEY:          /root/.ssh/id_rsa
HOSTTAG:          ip
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

However, when I manually trigger a crash via:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

The system reboots and no vmcore or .crash is ever written to /var/crash or to my remote SSH host. The SSH host never sees a login attempt. I see the trace output via ec2-get-console-output -r <instance>, and the system immediately reboots as shown above.

I'm very very stuck trying to debug it--everything seems correct, but no crash report.

Now, I'm not sure if this could possibly be related, but, ifquery is also crashing on startup, and there's no .crash report ever, and apport isn't aware that it crashed. I've yet to see apport ever actually create a .crash here. Could that possibly be what's going wrong with my crash dumps? Could anyone provide any insight into debugging this?

1

This may be related to this issue: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1421391

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.