39

I've got problem with Handbrake/ffmpeg. After ~5 minutes transcoding, the computer locks up. I'm fairly sure it's a kernel panic because caps-lock starts flashing.

There are a few logical questions about what to do and some about specific bugs but I'm really after one thing: what happened right before everything died?!

I've checked /var/log/kern.log and all I see around the time is me sticking in a DVD and then a few minutes later, the system booting up. No errors, no panic notice.

Is there any way to force panics to be logged? I'm fairly sure I can reproduce this (it's happened 100% of the times I've tried recently) so while I'd rather this "just worked", I'm happy enough to reboot a few times if it means I can find the cause of the panic.

3
  • Any specific message you get when transcoding? Might be useful in tracking down the solution ;)
    – Rinzwind
    Feb 16, 2012 at 15:29
  • @Rinzwind Nope. Didn't show anything, just froze.
    – Oli
    Feb 16, 2012 at 15:33
  • Most likely an overheating problem. Transcoding drives the CPU hard, and if your cooling isn't 100% effective the CPU will go into emergency shutdown. I've seen this happen when the thermal paste was dried out on the CPU heatsink, for example. It also happened when the overclocking settings were messed up in the BIOS. Try using xsensors to monitor the CPU temperature just before the lockup. Oct 20, 2015 at 22:10

4 Answers 4

31

If it really is a kernel panic then it won't be written into a log via normal methods. Since the kernel has at this point crashed, writing into the filesystem is a risky operation - not much of the kernel can be trusted anymore, so writes into logs might actually be spewing random crap over your bootloader!

Instead, you can dump the contents of memory into your swap, and then debug it later. This is known as a kernel crash / core dump.

The Ubuntu Wiki has a CrashdumpRecipe that may be useful - though it looks a bit out of date, I don't think too much should have changed.

2
26

All your system logs in Ubuntu are handled by rsyslog which keeps its configuration in /etc/rsyslog.conf and /etc/rsyslog.d/.

For more information on how to configure rsyslog and the possible options visit the rsyslog.conf man page.

Opening /etc/rsyslog.d/50-default.conf you can see that one of the lines contains

*.*;auth,authpriv.none -/var/log/syslog*

Meaning that the file you are looking for in this case is any of the huge /var/log/syslog logs you will probably have.

You can see that the file name also starts with a -, this means that the file is cached before writing, its great but can leave you with a bad log, what you want is that the log is written as soon as there is a problem. Remove the dash and reboot or reload rsyslog and then make your computer crash again, check /var/log/syslog.

3
  • 2
    removed that "-" rebooted, checked /var/log/syslog | grep panic. It did not work. Did I miss something ?
    – AAI
    Jan 2, 2018 at 20:01
  • Did something change in the recent kernel? I have a rule that says kern.* /var/log/kern.log but it seems like emergency messages do not have this kern.xxxx tag hence are not being captured.. What bothers me is that they are not there even in *.* file.
    – Sandeep
    Jun 1, 2021 at 0:40
  • This answer is actually incomplete, it should have more information about crash dumps. If a server crashes for different reasons, there wont be no logs, and the only way of capturing what happened is w a crash dump, either in a local FS, or somewhere remote ( normally NFS ). Jun 2, 2021 at 8:13
7

Serial port

The serial port is a simple low level communication mechanism between computers.

Advantages:

  • simple setup once (if you have the hardware)
  • reliable, since data transmission only depends on a simple wires and kernel API, which is less likely to be affected by the panic than say, the TCP/IP subsystem.

Downsides:

  • most modern laptops don't have the serial port anymore (exposed?) to save space. But desktops and virtual machines still do.
  • you also need a second computer with serial port as well to receive the data, but this is the case for basically all embedded development boards such as the Raspberry Pi.
  • limited by the length of the physical layer serial cable, unlike TCP/IP networks which are unlimited. This can however be worked around by plugging a device that converts between serial and TCP/IP on the serial of the remote computer

The serial port looks like this:

and on the RPI is available through the GPIO, here's what it looks like (with a USB interface to a laptop on the other side):

a uller example of that in use can be seen here.

Then, if you have the required hardware, connect from the second computer to the main computer with:

screen /dev/ttyS0 115200

This actually gives you a shell.

Then on the main machine, start the operation that panics.

When the panic happens, the panic dump is streamed over to the second machine, and you can see it all by scrolling up on the terminal.

Other methods

There are also other methods which overcome the hardware limitations mentioned above, at the cost of being more complex and less reliable. Notable methods:

  • netdump: streams the panic over TCP/IP. Relies on the TCP/IP subsystem not being corrupted.
  • kdump: appears to be the underlying mechanism of linux-crashdump mentioned at: https://askubuntu.com/a/104793/52975 Boots a second Linux kernel to examine the crashed kernel. What could possibly go wrong?! :-)

See also this great answer: https://unix.stackexchange.com/questions/60574/determining-cause-of-linux-kernel-panic

Step debugging

Ultimately, getting panic output requires that some kernel functionality work, and any kernel functionality could be corrupted by the panic.

But who needs panics if you can use GDB on the kernel? If you are that hardcore, have a look at:

Every problem falls once you have full visibility (and enough time!).

2
  • I don't think this works for Kernel panic on boot, right?
    – NH.
    Jan 14, 2021 at 22:37
  • 1
    @NH. it should work unless it is extremely early boot. I'm not sure 100% the exact point. And also have a look at earlyprintk/earlycon CLI args. Jan 14, 2021 at 22:59
0

On desktop distributions the kernel panic message will not be output, the console will. You can execute "sudo systemctl set-default multi-user.target " to disable the desktop and come to the console on the next reboot, so you can see the panic message. After confirming the type of error according to the panic message, such as "softlockup", you can set "kern.softlockup=1" and restart again, kexec/kdump can work normally. See https://stackoverflow.com/questions/64099710/where-does-the-linux-kernel-panic-message-go/64099932#64099932

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .