I'm having trouble debugging a system crash I'm experiencing.
Symptoms:
Without warning, my monitors turn blank and the system does not respond to any input. The computer cannot be communicated with over the network either. Both ping and ssh do not work. However, the computer is not "off". Its fans are still humming and its lights are still lit up. I've seen this crash happen anywhere from a few hours after boot, to a few days. I though the problem went away after 6 consecutive days of uptime, but then it crashed again. When this happens I have to press and hold the power button to force a hard reboot. Then it boots up fine and it seems like nothing is wrong (the temperature sensors all look good too). Furthermore, the problem does not seem to depend on CPU / GPU load. I'm able to run CPU intense processes without any issue. Its crashed on a few overnight jobs, but its also crashed on nights when the system was just idle.
In summary:
- Blank screen and no response to keyboard / mouse input
- No response to ssh / ping network input.
- Fans and LED lights are still on and buzzing. Holding the power button does force a reboot.
- I've seen time between crashes range from 2 hours to 144 hours. (with ~6-8 hours being the median time).
- Does not seem to depend CPU load. Crash happens regardless if the system is processing or idle.
System details
- The computer is about 7 years old.
- I'm on Ubuntu 16.04.1 LTS 64-bit (Gnome Version)
- The CPU is Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
- The GPU is an old
NVIDIA Corporation GT200b [GeForce GTX 285]
- PSU is 500W OCZ500MXSP
- 32 GB of memory (x4 8GB HyperX)
- A 128GB SSD
- 4 2TB HDDs configured in a software RAID10.
Output of lspci
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 05)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 2 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation P67 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
02:00.0 VGA compatible controller: NVIDIA Corporation GT200b [GeForce GTX 285] (rev a1)
04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
05:00.0 SATA controller: JMicron Technology Corp. JMB362 SATA Controller (rev 10)
06:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
08:03.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
Debugging steps taken:
I've setup my journalctl to be persistent so I can access logs from previous boots. However, looking through the logs I was unable to find anything that could point to an issue or a solution.
When I first saw the issue, my first instinct was CPU temperature.
I setup a system service to log the output of sensors
every 5 seconds.
The last few lines of that are here: https://gist.github.com/Erotemic/590699faa04956078f0ecc13f74217d8
Note the last line is full of null characters. I presume that the system crashed in the middle of writing to the file.
The temperatures all seem to be within reasonable ranges. One of the sensors CPUTIN: -60.0°C
has a constant value of -60, which I'm guessing means the sensor isn't working? The same seems to be true for a few of my fans. So, perhaps there is a temperature thing happening that I can't detect?
Last few lines of syslog bfore the latest crash: https://gist.github.com/Erotemic/4288a98be973edd8ed20915178122fdc
Its showing sensor alarms, but nothing seems (to me) to have a non-normal value.
The system is old, so it makes sense that maybe one of the hardware components is failing. However, I'm at a loss to figure out which one it is. The RAM was recently upgraded (the crash was happening before the upgrade), so I doubt its that. Any ideas on what it could be / how I could find out?