2

I recently upgraded the memory in my machine Ubuntu 16.04 from 4x8GB to 8x8GB. The retailer promised that the new memory would be compatible with my configuration, however I noticed that htop sometimes shows the full 64GB memory, sometimes only 48GB or even 16GB, different after each startup. The system freezes a couple of times a day. After one such freeze I had a look at the syslog:

Nov  7 13:08:09 embpc0032 kernel: [ 4524.820086] EDAC MC0: 7 CE memory read error on CPU_SrcID#0_Ha#0_Chan#3_DIMM#1 (channel:3 slot:1 page:0xb382e offset:0x8c0 grain:32 syndrome:0x0 -  OVERFLOW area:DRAM err_code:0001:0091 socket:0 ha:0 channel_mask:8 rank:4)
Nov  7 13:08:10 embpc0032 kernel: [ 4525.812100] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Nov  7 13:08:10 embpc0032 kernel: [ 4525.812107] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 7: cc000b0000010091
Nov  7 13:08:10 embpc0032 kernel: [ 4525.812110] EDAC sbridge MC0: TSC 0 
Nov  7 13:08:10 embpc0032 kernel: [ 4525.812112] EDAC sbridge MC0: ADDR b382fcc0 EDAC sbridge MC0: MISC 14022a286 
Nov  7 13:08:10 embpc0032 kernel: [ 4525.812117] EDAC sbridge MC0: PROCESSOR 0:306e4 TIME 1510056490 SOCKET 0 APIC 0
Nov  7 13:08:10 embpc0032 kernel: [ 4525.820084] EDAC MC0: 44 CE memory read error on CPU_SrcID#0_Ha#0_Chan#3_DIMM#1 (channel:3 slot:1 page:0xb382f offset:0xcc0 grain:32 syndrome:0x0 -  OVERFLOW area:DRAM err_code:0001:0091 socket:0 ha:0 channel_mask:8 rank:4)
Nov  7 13:08:11 embpc0032 kernel: [ 4526.812091] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Nov  7 13:08:11 embpc0032 kernel: [ 4526.812098] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 7: cc0001c000010091
Nov  7 13:08:11 embpc0032 kernel: [ 4526.812101] EDAC sbridge MC0: TSC 0 
Nov  7 13:08:11 embpc0032 kernel: [ 4526.812103] EDAC sbridge MC0: ADDR b382fcc0 EDAC sbridge MC0: MISC 214022a286 
Nov  7 13:08:11 embpc0032 kernel: [ 4526.812108] EDAC sbridge MC0: PROCESSOR 0:306e4 TIME 1510056491 SOCKET 0 APIC 0
Nov  7 13:08:11 embpc0032 kernel: [ 4526.820076] EDAC MC0: 7 CE memory read error on CPU_SrcID#0_Ha#0_Chan#3_DIMM#1 (channel:3 slot:1 page:0xb382f offset:0xcc0 grain:32 syndrome:0x0 -  OVERFLOW area:DRAM err_code:0001:0091 socket:0 ha:0 channel_mask:8 rank:4)
Nov  7 13:08:12 embpc0032 kernel: [ 4527.812083] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Nov  7 13:08:12 embpc0032 kernel: [ 4527.812091] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 7: cc00048000010091
Nov  7 13:08:12 embpc0032 kernel: [ 4527.812093] EDAC sbridge MC0: TSC 0 
Nov  7 13:08:12 embpc0032 kernel: [ 4527.812096] EDAC sbridge MC0: ADDR b382fcc0 EDAC sbridge MC0: MISC 14022a286 
Nov  7 13:08:12 embpc0032 kernel: [ 4527.812101] EDAC sbridge MC0: PROCESSOR 0:306e4 TIME 1510056492 SOCKET 0 APIC 0
Nov  7 13:08:12 embpc0032 kernel: [ 4527.820096] EDAC MC0: 18 CE memory read error on CPU_SrcID#0_Ha#0_Chan#3_DIMM#1 (channel:3 slot:1 page:0xb382f offset:0xcc0 grain:32 syndrome:0x0 -  OVERFLOW area:DRAM err_code:0001:0091 socket:0 ha:0 channel_mask:8 rank:4)
Nov  7 13:08:13 embpc0032 kernel: [ 4528.812100] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Nov  7 13:08:13 embpc0032 kernel: [ 4528.812108] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 7: cc0001c000010091
Nov  7 13:08:13 embpc0032 kernel: [ 4528.812110] EDAC sbridge MC0: TSC 0 
Nov  7 13:08:13 embpc0032 kernel: [ 4528.812112] EDAC sbridge MC0: ADDR b382fcc0 EDAC sbridge MC0: MISC 214022a286 
Nov  7 13:08:13 embpc0032 kernel: [ 4528.812117] EDAC sbridge MC0: PROCESSOR 0:306e4 TIME 1510056493 SOCKET 0 APIC 0

After this, a bunch of NULL symbols in the log, freeze and reboot. What could be the issue? What do channel and slot refer to in this context? It is a quad-channel mainboard (Fujitsu D3128-A2) equipped with DIMM.

E: I dug up the manual:

enter image description here

Is it safe to say that in the error log, channel:0 slot:0 would refer to what is called A1 in the manual? Because in the log I found around 4000 memory errors which were all in slot:1 across three channels, but never in slot:0. While all the RAM I bought new resides in the slots that end with 2 in the manual, so to me it looks like all errors stem from the new modules and not a single one from the old.

E: I came to work today and booted the computer. This was the output of lshw:

*-memory
      description: System Memory
      physical id: 1e
      slot: System board or motherboard
      size: 16GiB
    *-bank:0
         description: DIMM DDR3 800 MHz (1,2 ns)
         product: HMT41GR7AFR8C
         vendor: Hynix Semiconducto
         physical id: 0
         serial: 50404146
         slot: Node0_Dimm0
         size: 8GiB
         width: 64 bits
         clock: 800MHz (1.2ns)
    *-bank:1
         description: DIMM DDR3 800 MHz (1,2 ns)
         vendor: Undefined
         physical id: 1
         serial: 00000000
         slot: Node0_Dimm1
         size: 8GiB
         width: 64 bits
         clock: 800MHz (1.2ns)
    *-bank:2
         description: DIMM Synchronous [empty]
         product: Dimm2_PartNum
         vendor: Dimm2_Manufacturer
         physical id: 2
         serial: Dimm2_SerNum
         slot: Node0_Dimm2
         width: 64 bits
    *-bank:3
         description: DIMM Synchronous [empty]
         product: Dimm3_PartNum
         vendor: Dimm3_Manufacturer
         physical id: 3
         serial: Dimm3_SerNum
         slot: Node0_Dimm3
         width: 64 bits
    *-bank:4
         description: DIMM Synchronous [empty]
         product: Dimm4_PartNum
         vendor: Dimm4_Manufacturer
         physical id: 4
         serial: Dimm4_SerNum
         slot: Node0_Dimm4
         width: 64 bits
    *-bank:5
         description: DIMM Synchronous [empty]
         product: Dimm5_PartNum
         vendor: Dimm5_Manufacturer
         physical id: 5
         serial: Dimm5_SerNum
         slot: Node0_Dimm5
         width: 64 bits
    *-bank:6
         description: DIMM Synchronous [empty]
         product: Dimm6_PartNum
         vendor: Dimm6_Manufacturer
         physical id: 6
         serial: Dimm6_SerNum
         slot: Node0_Dimm6
         width: 64 bits
    *-bank:7
         description: DIMM Synchronous [empty]
         product: Dimm7_PartNum
         vendor: Dimm7_Manufacturer
         physical id: 7
         serial: Dimm7_SerNum
         slot: Node0_Dimm7
         width: 64 bits

After a reboot, this was the output of lshw:

*-memory
      description: System Memory
      physical id: 1e
      slot: System board or motherboard
      size: 48GiB
    *-bank:0
         description: DIMM DDR3 1866 MHz (0,5 ns)
         product: HMT41GR7AFR8C
         vendor: Hynix Semiconducto
         physical id: 0
         serial: 50404146
         slot: Node0_Dimm0
         size: 8GiB
         width: 64 bits
         clock: 1866MHz (0.5ns)
    *-bank:1
         description: DIMM DDR3 1866 MHz (0,5 ns)
         vendor: Undefined
         physical id: 1
         serial: 00000000
         slot: Node0_Dimm1
         size: 8GiB
         width: 64 bits
         clock: 1866MHz (0.5ns)
    *-bank:2
         description: DIMM Synchronous [empty]
         product: Dimm2_PartNum
         vendor: Dimm2_Manufacturer
         physical id: 2
         serial: Dimm2_SerNum
         slot: Node0_Dimm2
         width: 64 bits
    *-bank:3
         description: DIMM Synchronous [empty]
         product: Dimm3_PartNum
         vendor: Dimm3_Manufacturer
         physical id: 3
         serial: Dimm3_SerNum
         slot: Node0_Dimm3
         width: 64 bits
    *-bank:4
         description: DIMM DDR3 1866 MHz (0,5 ns)
         product: HMT41GR7AFR8C
         vendor: Hynix Semiconducto
         physical id: 4
         serial: 50404181
         slot: Node0_Dimm4
         size: 8GiB
         width: 64 bits
         clock: 1866MHz (0.5ns)
    *-bank:5
         description: DIMM DDR3 1866 MHz (0,5 ns)
         vendor: Undefined
         physical id: 5
         serial: 00000000
         slot: Node0_Dimm5
         size: 8GiB
         width: 64 bits
         clock: 1866MHz (0.5ns)
    *-bank:6
         description: DIMM DDR3 1866 MHz (0,5 ns)
         product: HMT41GR7AFR8C
         vendor: Hynix Semiconducto
         physical id: 6
         serial: 50404153
         slot: Node0_Dimm6
         size: 8GiB
         width: 64 bits
         clock: 1866MHz (0.5ns)
    *-bank:7
         description: DIMM DDR3 1866 MHz (0,5 ns)
         vendor: Undefined
         physical id: 7
         serial: 00000000
         slot: Node0_Dimm7
         size: 8GiB
         width: 64 bits
         clock: 1866MHz (0.5ns)

Note how the first time, the two modules that are recognized are listed with different stats than after the reboot (they are in fact 1866 MHz).

4

To troubleshoot this problem...

  1. first reseat all of the memory modules
  2. run the free memtest86.com memory test
  3. reconfigure the memory modules into the correct slots
  4. retest memtest86 test

Reseat:

  • power off the computer
  • touch the metal chassis to dissipate any static charge
  • remove the AC power cord
  • depress the power switch to dissipate any charge left in the power supply
  • remove and reseat ALL memory modules

Memtest86:

  • go to memtest86.com and download the free memory test
  • run at least one complete pass, more if you have the time
  • if you get a failure, then start removing 2 memory modules at a time and retest
  • if you don't get a failure, read the next section on memory configuration

Configuration:

Memory interleaving is a modern technique to speed up memory access. It requires that the memory be configured using equal pairs of memory modules. Your high-end system appears to have 4 memory channels... A/B/C/D.

Take your original 4 memory modules and fill all the module 1 positions first, then use the 4 new memory modules and fill the remaining module 2 positions.

Rerun the memtest86 test.

  • Thanks. What do you mean exactly by reseat? You might not have seen my latest edit--all errors in today's syslog appeared in slot:1 across three channels, never slot:0 though. I suspect that slot:0 is where the old memory sits. I tried to run memtest86 last night but wasted a DVD and a USB stick after the machine froze while making them bootable. Hoping to try again today – smcs Nov 7 '17 at 14:01
  • 1
    @speedymcs to reseat, I mean fully/partially remove a memory module and reinsert it. That takes care of bad/dirty connections in the slot. If you're having trouble making a memtest disc, remove the new memory and then build the disc. Assuming that the new memory modules are correct for your machine, and match the specs of the original memory modules, I suspect that the "Configuration" section of my answer will fix your problem. – heynnema Nov 7 '17 at 14:07
  • Thanks. The configuration is definitely correct. The old modules all sit in module 1 slots which are also all the same color. It would all be so much easier if I was actually allowed to touch the damn thing ha. – smcs Nov 7 '17 at 14:15
  • @speedymcs as long as you ground yourself first, and remove the AC power, as I instruct, you should be able to touch any of the memory modules and reseat/reconfigure them at your desire. After grounding yourself, limit your movements to reduce the static electricity buildup. Report back. – heynnema Nov 7 '17 at 14:21
  • Sorry, what I meant is that I'm not allowed to change any hardware configuration according to company policy; an IT colleague needs to do this and of course they are always busy ha. I ran Memtest86 last night without reinserting the modules, each test four times, 14 hours in total. It did not find a single error. So the next and probably last thing to try is to have IT remove all modules and clean the slots with dry air or something...? – smcs Nov 8 '17 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.