30

I've got problem with Handbrake/ffmpeg. After ~5 minutes transcoding, the computer locks up. I'm fairly sure it's a kernel panic because caps-lock starts flashing.

There are a few logical questions about what to do and some about specific bugs but I'm really after one thing: what happened right before everything died?!

I've checked /var/log/kern.log and all I see around the time is me sticking in a DVD and then a few minutes later, the system booting up. No errors, no panic notice.

Is there any way to force panics to be logged? I'm fairly sure I can reproduce this (it's happened 100% of the times I've tried recently) so while I'd rather this "just worked", I'm happy enough to reboot a few times if it means I can find the cause of the panic.

  • Any specific message you get when transcoding? Might be useful in tracking down the solution ;) – Rinzwind Feb 16 '12 at 15:29
  • @Rinzwind Nope. Didn't show anything, just froze. – Oli Feb 16 '12 at 15:33
  • Most likely an overheating problem. Transcoding drives the CPU hard, and if your cooling isn't 100% effective the CPU will go into emergency shutdown. I've seen this happen when the thermal paste was dried out on the CPU heatsink, for example. It also happened when the overclocking settings were messed up in the BIOS. Try using xsensors to monitor the CPU temperature just before the lockup. – Neil Mayhew Oct 20 '15 at 22:10
21

All your system logs in Ubuntu are handled by rsyslog which keeps its configuration in /etc/rsyslog.conf and /etc/rsyslog.d/.

For more information on how to configure rsyslog and the possible options visit the rsyslog.conf man page.

Opening /etc/rsyslog.d/50-default.conf you can see that one of the lines contains

*.*;auth,authpriv.none -/var/log/syslog*

Meaning that the file you are looking for in this case is any of the huge /var/log/syslog logs you will probably have.

You can see that the file name also starts with a -, this means that the file is cached before writing, its great but can leave you with a bad log, what you want is that the log is written as soon as there is a problem. Remove the dash and reboot or reload rsyslog and then make your computer crash again, check /var/log/syslog.

  • 1
    removed that "-" rebooted, checked /var/log/syslog | grep panic. It did not work. Did I miss something ? – AAI Jan 2 '18 at 20:01
26

If it really is a kernel panic then it won't be written into a log via normal methods. Since the kernel has at this point crashed, writing into the filesystem is a risky operation - not much of the kernel can be trusted anymore, so writes into logs might actually be spewing random crap over your bootloader!

Instead, you can dump the contents of memory into your swap, and then debug it later. This is known as a kernel crash / core dump.

The Ubuntu Wiki has a CrashdumpRecipe that may be useful - though it looks a bit out of date, I don't think too much should have changed.

  • 10
    The CrashdumpRecipe refers to the Linux Kernel Crash Dump (LKCD) tool available on Sourceforge - there is a package for Ubuntu called linux-crashdump; this package is still available in all versions. – Mei Mar 29 '12 at 14:02
3

Serial port

The serial port is a simple low level communication mechanism between computers.

Advantages:

  • simple setup once (if you have the hardware)
  • reliable, since data transmission only depends on a simple wires and kernel API, which is less likely to be affected by the panic than say, the TCP/IP subsystem.

Downsides:

  • most modern laptops don't have the serial port anymore (exposed?) to save space. But desktops and virtual machines still do.
  • you also need a second computer with serial port as well to receive the data, but this is the case for basically all embedded development boards such as the Raspberry Pi.
  • limited by the length of the physical layer serial cable, unlike TCP/IP networks which are unlimited. This can however be worked around with a device that interfaces between serial and TCP/IP. But there are devices which convert between the two.

The serial port looks like this:

and on the RPI is available through the GPIO.

Then, if you have the required hardware, connect from the second computer to the main computer with:

screen /dev/ttyS0 115200

This actually gives you a shell.

Then on the main machine, start the operation that panics.

When the panic happens, the panic dump is streamed over to the second machine, and you can see it all by scrolling up on the terminal.

Other methods

There are also other methods which overcome the hardware limitations mentioned above, at the cost of being more complex and less reliable. Notable methods:

  • netdump: streams the panic over TCP/IP. Relies on the TCP/IP subsystem not being corrupted.
  • kdump: appears to be the underlying mechanism of linux-crashdump mentioned at: https://askubuntu.com/a/104793/52975 Boots a second Linux kernel to examine the crashed kernel. What could possibly go wrong?! :-)

See also this great answer: https://unix.stackexchange.com/questions/60574/determining-cause-of-linux-kernel-panic

Step debugging

Ultimately, getting panic output requires that some kernel functionality work, and any kernel functionality could be corrupted by the panic.

But who needs panics if you can use GDB on the kernel? If you are that hardcore, have a look at:

Every problem falls once you have full visibility (and enough time!).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.