3

I'm trying to get kernel crashdump to work properly in Ubuntu 15.10 with the stock 4.2.0-22-generic kernel. I have followed the methods described here and here precisely. But, when I manually trigger a crash via:

echo c | sudo tee /proc/sysrq-trigger

the system crashes, and reboots, but no crash output is saved in /var/crash.

As this is EC2, I don't have a read/write console--I can only get read-only console output, and I'm not seeing much useful output:

[  473.666303] sysrq: SysRq : Trigger a crash
[  473.668278] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  473.671624] IP: [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[  473.672244] PGD 3e235c067 PUD 3e2351067 PMD 0
[  473.672244] Oops: 0002 [#1] SMP
[  473.672244] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev xen_fbfront intel_rapl fb_sys_fops iosf_mbi input_leds serio_raw parport_pc 8250_fintek i2c_piix4 parport mac_hid autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea aesni_intel aes_x86_64 sysfillrect lrw sysimgblt gf128mul ttm glue_helper ablk_helper drm_kms_helper cryptd psmouse drm ixgbevf pata_acpi floppy
[  473.672244] CPU: 3 PID: 2814 Comm: bash Not tainted 4.2.0-22-generic #27-Ubuntu
[  473.672244] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
[  473.672244] task: ffff8803d1d86e00 ti: ffff8803dc414000 task.ti: ffff8803dc414000
[  473.672244] RIP: 0010:[<ffffffff814c79e6>]  [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[  473.672244] RSP: 0018:ffff8803dc417e28  EFLAGS: 00010246
[  473.672244] RAX: 000000000000000f RBX: 0000000000000063 RCX: 0000000000000000
[  473.672244] RDX: 0000000000000000 RSI: ffff8803ff2ce938 RDI: 0000000000000063
[  473.672244] RBP: ffff8803dc417e28 R08: 0000000000000002 R09: 000000000000024d
[  473.672244] R10: 000000000000a614 R11: 000000000000024d R12: 0000000000000004
[  473.672244] R13: 0000000000000000 R14: ffffffff81cb48e0 R15: 0000000000000000
[  473.672244] FS:  00007fecdb3ca700(0000) GS:ffff8803ff2c0000(0000) knlGS:0000000000000000
[  473.672244] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  473.672244] CR2: 0000000000000000 CR3: 00000003e234b000 CR4: 00000000001406e0
[  473.672244] Stack:
[  473.672244]  ffff8803dc417e58 ffffffff814c821a 0000000000000002 fffffffffffffffb
[  473.672244]  ffff8803dc417f18 0000000000000002 ffff8803dc417e78 ffffffff814c86a3
[  473.672244]  0000000000000002 ffff8803f816c900 ffff8803dc417e98 ffffffff81266aa2
[  473.672244] Call Trace:
[  473.672244]  [<ffffffff814c821a>] __handle_sysrq+0xea/0x140
[  473.672244]  [<ffffffff814c86a3>] write_sysrq_trigger+0x33/0x40
[  473.672244]  [<ffffffff81266aa2>] proc_reg_write+0x42/0x70
[  473.672244]  [<ffffffff811fca68>] __vfs_write+0x18/0x40
[  473.672244]  [<ffffffff811fd3f6>] vfs_write+0xa6/0x1a0
[  473.672244]  [<ffffffff810c3e21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[  473.672244]  [<ffffffff811fe0e5>] SyS_write+0x55/0xc0
[  473.672244]  [<ffffffff8121b31f>] ? __close_fd+0x8f/0xb0
[  473.672244]  [<ffffffff817f02b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[  473.672244] Code: 45 3b 7d 34 75 e5 4c 89 ef e8 f7 f7 ff ff eb db 0f 1f 44 00 00 0f 1f 44 00 00 55 c7 05 a8 74 a2 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 48 89 e5 53 8d
[  473.672244] RIP  [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[  473.672244]  RSP <ffff8803dc417e28>
[  473.672244] CR2: 0000000000000000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.2.0-22-generic (buildd@lcy01-22) (gcc version 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) ) #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 (Ubuntu 4.2.0-22.27-generic 4.2.6)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 crashkernel=384M

...
[    3.021894] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
...
[  OK  ] Started memcached daemon.
         Starting LSB: Execute the kexec -e command to reboot system...
...
[  OK  ] Started LSB: Record successful boot for GRUB.
[  OK  ] Started LSB: automatic crash report generation.
[  OK  ] Started LSB: Execute the kexec -e command to reboot system.
...
[  OK  ] Started LSB: Load kernel image with kexec.
ondemand.service
rc-local.service
grub-common.service
         Stopping LSB: Start NTP daemon...
apport.service
kexec.service
...
lxc.service
[  OK  ] Started LXC Container Initialization and Autoboot Code.
         Starting Container hypervisor based on LXC - boot time check...
[   34.181647] kdump-tools[773]: Starting kdump-tools:  * loaded kdump kernel
kdump-tools.service
[  OK  ] Started Kernel crash dump capture service.
[  OK  ] Started Container hypervisor based on LXC - boot time check.

And then the system is fully back online, with nothing in /var/crash except .lock and kexec_cmd.

I've tried crashkernel=128M, crashkernel=256M, crashkernel=384M, 512M, 256@0, 256@16M, etc.

I even tried enabling SSH in /etc/default/grub.d/kexec-tools.cfg, with a machine I've verified I can access from this machine, with a manual SSH_KEY configured that exists, works, and has proper permissions set, but the remote machine doesn't show a connection attempt at all.

The output of kdump-config show looks proper:

DUMP_MODE:        kdump
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 0x2c000000
SSH:              [email protected]
SSH_KEY:          /root/.ssh/id_rsa
HOSTTAG:          ip
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

However, when I manually trigger a crash via:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

The system reboots and no vmcore or .crash is ever written to /var/crash or to my remote SSH host. The SSH host never sees a login attempt. I see the trace output via ec2-get-console-output -r <instance>, and the system immediately reboots as shown above.

I'm very very stuck trying to debug it--everything seems correct, but no crash report.

Now, I'm not sure if this could possibly be related, but, ifquery is also crashing on startup, and there's no .crash report ever, and apport isn't aware that it crashed. I've yet to see apport ever actually create a .crash here. Could that possibly be what's going wrong with my crash dumps? Could anyone provide any insight into debugging this?

2 Answers 2

1

This may be related to this issue: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1421391

0

Just an idea - try disabling some of the most memory intensive modules that are unrelated to the kdump collection, I've seen many high-performance networking drivers to cause OOMs at work, also my home machine does have a high-end graphics, both examples lead to a lot of memory being loaded on kdump, leading to memory shortage, after all the reserved kdump memory is a tiny fraction of the installed RAM on the host as it is reserved on boot and unavailable afterwards.

So to identify which modules consume a lot of memory:

 $ lsmod | sort -nk2 -r | head
amdgpu               4116480  16
btrfs                1228800  2
kvm                   655360  0
nfsv4                 638976  2
drm                   487424  8 gpu_sched,drm_kms_helper,amdgpu,ttm
sunrpc                380928  9 nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs
aesni_intel           372736  0
fscache               368640  2 nfsv4,nfs
nfs                   299008  2 nfsv4
igb                   221184  0

In my case the amdgpu is on the top, but you could have all the modules I'm stumbling upon at work, like ixgbe, i40e, mlx5_core, etc.

To disable these just for the kdump kernel edit the /etc/default/kdump-tools and uncomment (perhaps copy, then uncomment) the KDUMP_CMDLINE_APPEND, then add the drivers to be blacklisted. Some might be in the kernel, some in the initrd, so to be sure, add each as $driver_name.blacklist=1 and rd.driver.blacklist=$driver_name on this line, this is an example with amdgpu below:

[snip]
#KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0"
KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0 amdgpu.blacklist=1 rd.driver.blacklist=amdgpu"
[snip]

Then just reload the kdump-tools and ensure the new configuration is loaded:

$ sudo systemctl restart kdump-tools
$ kdump-config show
DUMP_MODE:        kdump
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 0x
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.3.0-40-lowlatency
kdump initrd: 
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.3.0-40-lowlatency
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/@/boot/vmlinuz-5.3.0-40-lowlatency root=UUID=a745358b-a4e6-4a16-a347-5fa3d65e78a7 ro rootflags=subvol=@ quiet splash vt.handoff=1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0 amdgpu.blacklist=1 rd.driver.blacklist=amdgpu" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

Then retry collecting.

Cheers, T.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .