The serial port is a simple low level communication mechanism between computers.
- simple setup once (if you have the hardware)
- reliable, since data transmission only depends on a simple wires and kernel API, which is less likely to be affected by the panic than say, the TCP/IP subsystem.
- most modern laptops don't have the serial port anymore (exposed?) to save space. But desktops and virtual machines still do.
- you also need a second computer with serial port as well to receive the data, but this is the case for basically all embedded development boards such as the Raspberry Pi.
- limited by the length of the physical layer serial cable, unlike TCP/IP networks which are unlimited. This can however be worked around with a device that interfaces between serial and TCP/IP. But there are devices which convert between the two.
The serial port looks like this:
and on the RPI is available through the GPIO.
Then, if you have the required hardware, connect from the second computer to the main computer with:
screen /dev/ttyS0 115200
This actually gives you a shell.
Then on the main machine, start the operation that panics.
When the panic happens, the panic dump is streamed over to the second machine, and you can see it all by scrolling up on the terminal.
There are also other methods which overcome the hardware limitations mentioned above, at the cost of being more complex and less reliable. Notable methods:
- netdump: streams the panic over TCP/IP. Relies on the TCP/IP subsystem not being corrupted.
- kdump: appears to be the underlying mechanism of linux-crashdump mentioned at: https://askubuntu.com/a/104793/52975 Boots a second Linux kernel to examine the crashed kernel. What could possibly go wrong?! :-)
See also this great answer: https://unix.stackexchange.com/questions/60574/determining-cause-of-linux-kernel-panic
Ultimately, getting panic output requires that some kernel functionality work, and any kernel functionality could be corrupted by the panic.
But who needs panics if you can use GDB on the kernel? If you are that hardcore, have a look at:
Every problem falls once you have full visibility (and enough time!).