Where are kernel panic logs?

Question

I've got problem with Handbrake/ffmpeg. After ~5 minutes transcoding, the computer locks up. I'm fairly sure it's a kernel panic because caps-lock starts flashing.

There are a few logical questions about what to do and some about specific bugs but I'm really after one thing: what happened right before everything died?!

I've checked /var/log/kern.log and all I see around the time is me sticking in a DVD and then a few minutes later, the system booting up. No errors, no panic notice.

Is there any way to force panics to be logged? I'm fairly sure I can reproduce this (it's happened 100% of the times I've tried recently) so while I'd rather this "just worked", I'm happy enough to reboot a few times if it means I can find the cause of the panic.

Answer 1

If it really is a kernel panic then it won't be written into a log via normal methods. Since the kernel has at this point crashed, writing into the filesystem is a risky operation - not much of the kernel can be trusted anymore, so writes into logs might actually be spewing random crap over your bootloader!

Instead, you can dump the contents of memory into your swap, and then debug it later. This is known as a kernel crash / core dump.

The Ubuntu Wiki has a CrashdumpRecipe that may be useful - though it looks a bit out of date, I don't think too much should have changed.

Answer 2

All your system logs in Ubuntu are handled by rsyslog which keeps its configuration in /etc/rsyslog.conf and /etc/rsyslog.d/.

For more information on how to configure rsyslog and the possible options visit the rsyslog.conf man page.

Opening /etc/rsyslog.d/50-default.conf you can see that one of the lines contains

*.*;auth,authpriv.none -/var/log/syslog*

Meaning that the file you are looking for in this case is any of the huge /var/log/syslog logs you will probably have.

You can see that the file name also starts with a -, this means that the file is cached before writing, its great but can leave you with a bad log, what you want is that the log is written as soon as there is a problem. Remove the dash and reboot or reload rsyslog and then make your computer crash again, check /var/log/syslog.

Answer 3

Serial port

The serial port is a simple low level communication mechanism between computers.

Advantages:

• simple setup once (if you have the hardware)
• reliable, since data transmission only depends on a simple wires and kernel API, which is less likely to be affected by the panic than say, the TCP/IP subsystem.

Downsides:

• most modern laptops don't have the serial port anymore (exposed?) to save space. But desktops and virtual machines still do.
• you also need a second computer with serial port as well to receive the data, but this is the case for basically all embedded development boards such as the Raspberry Pi.
• limited by the length of the physical layer serial cable, unlike TCP/IP networks which are unlimited. This can however be worked around by plugging a device that converts between serial and TCP/IP on the serial of the remote computer

The serial port looks like this:

and on the RPI is available through the GPIO, here's what it looks like (with a USB interface to a laptop on the other side):

a uller example of that in use can be seen here.

Then, if you have the required hardware, connect from the second computer to the main computer with:

screen /dev/ttyS0 115200

This actually gives you a shell.

Then on the main machine, start the operation that panics.

When the panic happens, the panic dump is streamed over to the second machine, and you can see it all by scrolling up on the terminal.

Other methods

There are also other methods which overcome the hardware limitations mentioned above, at the cost of being more complex and less reliable. Notable methods:

• netdump: streams the panic over TCP/IP. Relies on the TCP/IP subsystem not being corrupted.
• kdump: appears to be the underlying mechanism of linux-crashdump mentioned at: https://askubuntu.com/a/104793/52975 Boots a second Linux kernel to examine the crashed kernel. What could possibly go wrong?! :-)

See also this great answer: https://unix.stackexchange.com/questions/60574/determining-cause-of-linux-kernel-panic

Step debugging

Ultimately, getting panic output requires that some kernel functionality work, and any kernel functionality could be corrupted by the panic.

But who needs panics if you can use GDB on the kernel? If you are that hardcore, have a look at:

• QEMU or other VMs: https://stackoverflow.com/a/42316607/895245
• JTAG

Every problem falls once you have full visibility (and enough time!).

Answer 4

On desktop distributions the kernel panic message will not be output, the console will. You can execute "sudo systemctl set-default multi-user.target " to disable the desktop and come to the console on the next reboot, so you can see the panic message. After confirming the type of error according to the panic message, such as "softlockup", you can set "kern.softlockup=1" and restart again, kexec/kdump can work normally. See https://stackoverflow.com/questions/64099710/where-does-the-linux-kernel-panic-message-go/64099932#64099932