kernel-panic after large file copy [closed]

Question

I'm running ubuntu-server, 3.2.0-29-generic-pae. Not every time, but enough that it is reproducible, when I cp or mv a large file the kernel panics. I read that changing the scheduler sometimes helps so I'm currently using noop instead of cfq. I've run memtest with no discernible results. And run badblocks on my ide that hosts the operating system. I have two SATA 500gb hard drives attached as well. The crash that is currently documented occurred when transferring from one to the other.

I ran cat /proc/kmsg on an ssh terminal and simulated a crash on my server, giving the following results:

7t:(ul
<4>[ 7486.965231] Disabling lock debugging due to kernel taint
<0>[ 7486.965231] [Hardware Error]: CPU 1: Machine Check Exception: 4 Bank 0: b20000001040080f
<0>[ 7486.965231] [Hardware Error]: RIP !INEXACT! 00:<00000000c1018b5c>
<0>[ 7486.965231] [Hardware Error]: TSC 1649e27da970
<0>[ 7486.965231] [Hardware Error]: PROCESSOR 0:f34 TIME 1358822531 SOCKET 0 APIC 1 microcode 13
<0>[ 7486.965231] [Hardware Error]: Run the above through 'mcelog --ascii'
<0>[ 7486.965231] [Hardware Error]: CPU 0: Machine Check Exception: 4 Bank 0: b20000001040080f
<0>[ 7486.965231] [Hardware Error]: RIP !INEXACT! 00:<00000000c12b7906>
<0>[ 7486.965231] [Hardware Error]: TSC 1649e27daae0
<0>[ 7486.965231] [Hardware Error]: PROCESSOR 0:f34 TIME 1358822531 SOCKET 0 APIC 0 microcode 13
<0>[ 7486.965231] [Hardware Error]: Run the above through 'mcelog --ascii'
<0>[ 7486.965231] [Hardware Error]: Machine check: Processor context corrupt
<0>[ 7486.965231] Kernel panic - not syncing: Fatal Machine check
<4>[ 7486.965231] Pid: 0, comm: swapper/1 Tainted: G   M         3.2.0-29-generic-pae #46-Ubuntu
<4>[ 7486.965231] Call Trace:
<4>[ 7486.965231]  [<c158f812>] ? printk+0x2d/0x2f
<4>[ 7486.965231]  [<c158f6e0>] panic+0x5c/0x161
<4>[ 7486.965231]  [<c10247ef>] mce_panic.part.14+0x13f/0x170
<4>[ 7486.965231]  [<c1024872>] mce_panic+0x52/0x90
<4>[ 7486.965231]  [<c1024a18>] mce_reign+0x168/0x170
<4>[ 7486.965231]  [<c1024bb5>] mce_end+0x105/0x110
<4>[ 7486.965231]  [<c10252db>] do_machine_check+0x32b/0x4f0
<4>[ 7486.965231]  [<c1018b5c>] ? mwait_idle+0x7c/0x1d0
<4>[ 7486.965231]  [<c1024fb0>] ? mce_log+0x120/0x120
<4>[ 7486.965231]  [<c15a5e47>] error_code+0x67/0x6c
<4>[ 7486.965231]  [<c1018b5c>] ? mwait_idle+0x7c/0x1d0
<4>[ 7486.965231]  [<c1010338>] cpu_idle+0xa8/0xe0
<4>[ 7486.965231]  [<c1588923>] start_secondary+0xe8/0xed
<3>[ 7486.965231] panic occurred, switching back to text console

Thanks for you help! curios

P.S. this is my first post, so I apologize in advance for any breaks of etiquette.

Answer 1

It looks like you've lost some of the log output, but there's some data there that seems to be pertinent:

[ 7486.965231] [Hardware Error]: CPU 1: Machine Check Exception: 4 Bank 0: b20000001040080f
[ 7486.965231] [Hardware Error]: RIP !INEXACT! 00:<00000000c1018b5c>
[ 7486.965231] [Hardware Error]: TSC 1649e27da970
[ 7486.965231] [Hardware Error]: PROCESSOR 0:f34 TIME 1358822531 SOCKET 0 APIC 1 microcode 13

This indicates that a "Machine Check Exception" was reported by the processor; this is generally due hardware issues.

The error code itself (b20000001040080f) seems to indicate an error with the bus or interconnect, perhaps you have a misbehaving PCI device? Can you remove devices and re-test, to see if the results differ?

My guess would be that it's an issue involving the disk controller. To test this hypothesis, we could try to reproduce the problem with and without load on the controller:

• Can you reproduce the problem with only disk IO, and less memory pressure?
  • Try something like: dd if=/dev/sda of=/dev/null bs=4096,
  • Or, if you have a disk device that you can overwrite: dd if=/dev/zero of=/dev/sdc bs=4096 - but note that this will erase all data on the specified device (/dev/sdc in this example).
• Can you reproduce this without involving disk IO?
  • You mention you've tried memtest; perhaps try a userspace memory test, like memtester, and keep an eye on dmesg for any MCEs reported?

If it does seem to be only occuring when loading the disk controller, you could then try to reproduce the problem using a different PCI device. Perhaps by putting a lot of data through the network device? This may give you some data on whether the problem is in the device itself, or the bus.

Because this looks like a hardware issue, I'd avoid filing a bug until you've done some more digging.