Serious RAID problem that panics system

Question

System: Dual Xeon E5-2630 CPU, 32GB RAM, Primary disk is a SATA-III 512GB Crucial SSD OS: Xubuntu 14.04.1

I am having a serious problem with RAID on this new system and hope some of you can provide some insight. Currently, the primary SSD with the root filesystem is not mirrored, although I plan to mirror it to a second identical SSD in the future. I am attempting to set up a RAID on a secondary HDD set and am unwilling to upgrade the primary SSD set until this problem is solved.

I have a pair of SATA-III Seagate ST4000DM0004TB Baracuda 4TB drives in this system which were formatted identically with a single large ext4 GPT partition. I have been attempting to create a useful RAID 1 mirror on these disks which is then mounted on /x. At one point I had something that appeared to be stable and ran for a few weeks until I tried to modify the Array, at which point it failed. Every time the mirror fails, it apparently panics the system and the root filesystem on the SSD is remounted read-only as per the setting in /etc/fstab (errors=remount-ro). Of course the system is now useless and requires a hard reset. The system reboots but the mirror is now totally corrupted and must usually be destroyed and rebuilt. I've run hardware diagnostics and see no problem. There are zero hints as to what is wrong in any of the log files (dmesg, kern.log, syslog). Here are some details:

---

I create the Array as follows:

# mdadm --create /dev/md2 --verbose --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
mdadm: /dev/sdc1 appears to contain an ext2fs file system
    size=-387950592K mtime=Wed Dec 31 16:00:00 1969
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device. If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: /dev/sdd1 appears to contain an ext2fs file system
    size=-387950592K mtime=Wed Dec 31 16:00:00 1969
mdadm: size set to 3906885440K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.

I check the RAID build progress:

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sdd1[1] sdc1[0]
    3906885440 blocks super 1.2 [2/2] [UU]
    [>....................] resync = 0.4% (17314560/3906885440) finish=415.6min speed=155968K/sec

unused devices: <none>

I continue to periodically monitor the resync operation with the above command and it moves along without problem. However, at some point (I've had the resync get to anywhere from 4% to 60% synced), the system panics and root is remounted RO. When the system is rebooted, I usually find the following:

# l /dev/md*
/dev/md127 /dev/md127p1

/dev/md:
dymaxion:2@ dymaxion:2p1@

In the case where I did manage to get /dev/md2 built and running, I had /dev/md2 and /dev/md2p1 devices with nothing in the /dev/md subdirectory. Here the panicked system seems to try to salvage the array as md127. I do not understand why, but this has happened repeatedly. Possibly it is the result of some algorithm coded into the mdadm software.

Sometime the md127 array is degraded to such a point that it cannot be mounted at boot time (there is an entry for the array in /etc/fstab) and other times it does mount and attempt to resync. However, it often panics the system during this operation, leading to a continual series of reboots.

I then destroy the array and attempt to recreate it. These are the commands I use to destroy it.

# mdadm --stop /dev/md127
mdadm: stopped /dev/md127

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

unused devices:

# mdadm -QD /dev/md127
mdadm: cannot open /dev/md127: No such file or directory

# mdadm --zero-superblock /dev/sdc1
# mdadm --zero-superblock /dev/sdd1

I have tried building the array on a quiet system with no desktop processes running other than a few terminal windows. I've run the checkbox-gui hardware test suite and everything checks out fine. I tried unplugging all other SATA disks, USB ports, Card Reader, Optical Disk, etc. and then run the build and it still fails.

Can anyone spot some reason why the array is failing or suggest some way to better determine what is happening?

Here is some additional information on my efforts.

What I have been doing on my Sun Server (Solaris 10) for the past 10 years is attach a third disk to the array, allow it to sync up, detach it from the array and then take it off site for disaster recovery. This has been working great and this is what I planned to do on this Ubuntu system.

Using the above procedures, I did once manage to get /dev/md2 properly built with the two internal disks. The system ran without problem for a few weeks, so I was ready to attach the third disk using a hot-swap bay. I rebooted with the 3rd disk in the hot-swap bay. Because of the arbitrary device reassignments, the new disk appeared as /dev/sda and the mirror was using /dev/sdd and /dev/sde.

# mdadm -QD /dev/md2p1 (or: # mdadm -QD /dev/md2)
/dev/md2:
        Version : 1.2
  Creation Time : Tue Sep 9 17:50:52 2014
     Raid Level : raid1
     Array Size : 3906885440 (3725.90 GiB 4000.65 GB)
  Used Dev Size : 3906885440 (3725.90 GiB 4000.65 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Fri Sep 19 14:02:45 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : dymaxion:2 (local to host dymaxion)
           UUID : 1e131e20:9c899b31:a7494bc5:1dbc253f
         Events : 129

      Number Major Minor RaidDevice State
         3 8 65 0 active sync /dev/sde1
         2 8 49 1 active sync /dev/sdd1


# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sdd1[2] sde1[3]
      3906885440 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Everything looks good. Let's add /dev/sda1 as a spare to /dev/md2p1:

# mdadm /dev/md2 --add /dev/sda1

# mdadm -QD /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Tue Sep 9 17:50:52 2014
     Raid Level : raid1
     Array Size : 3906885440 (3725.90 GiB 4000.65 GB)
  Used Dev Size : 3906885440 (3725.90 GiB 4000.65 GB)
   Raid Devices : 2
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Oct 17 13:36:13 2014
          State : clean
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

           Name : dymaxion:2 (local to host dymaxion)
           UUID : 1e131e20:9c899b31:a7494bc5:1dbc253f
         Events : 130

      Number Major Minor RaidDevice State
         3 8 65 0 active sync /dev/sde1
         2 8 49 1 active sync /dev/sdd1

         4 8 1 - spare /dev/sda1

OK, let's attach the spare to the array using the grow option:

# mdadm /dev/md2 --grow -n3

# mdadm -QD /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Tue Sep 9 17:50:52 2014
     Raid Level : raid1
     Array Size : 3906885440 (3725.90 GiB 4000.65 GB)
  Used Dev Size : 3906885440 (3725.90 GiB 4000.65 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Oct 17 14:43:08 2014
          State : clean, degraded, recovering
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           Name : dymaxion:2 (local to host dymaxion)
           UUID : 1e131e20:9c899b31:a7494bc5:1dbc253f
         Events : 134

      Number Major Minor RaidDevice State
         3 8 65 0 active sync /dev/sde1
         2 8 49 1 active sync /dev/sdd1
         4 8 1 2 spare rebuilding /dev/sda1

Looks good! Allow the third disk to sync:

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda1[4] sdd1[2] sde1[3]
      3906885440 blocks super 1.2 [3/2] [UU_]
      [>....................] recovery = 0.7% (27891328/3906885440) finish=376.2min speed=171823K/sec

unused devices: <none>

Somewhere after the mirror had synced more than 10%, the system panicked. This time, when the system was rebooted, the boot process was unable to reattach the mirror to /x and prompted to retry or skip the mount. I skipped it and when the system booted, there was no way to reactivate /dev/md2. Ultimately I had to destroy it and start over. I never got this close again. had this worked, the plan was to mark the third disk as failed, remove it and grow the array back to two devices (or two devices and a missing spare.)

Do you see anything wrong with any of this build procedure?

I apologize for the long post. I wanted to try and provide as much information as possible to try and anticipate any questions.

Any suggestions are greatly appreciated. I am particularly concerned about what is causing the system to panic.

---

Everything below here was added Saturday, November 15, 2014

---

First, let me clarify an apparent misunderstanding. @psusi wrote:

Since you didn't mention creating a filesystem on the raid array and mounting it after creating the array, and mdadm warned you that /dev/sdc1 already has an ext2 filesystem in it, I'm guessing you mean you already have a filesystem in /dev/sdc1, and that is what is being remounted read only.

No. The root filesystem is on its own solid state SATA-III drive (sda1) while I am attempting to build the md2 mirror using two other 4TB disks (sdc and sdd). It is while this mirror is syncing that something goes wrong, the entire system panics, and it is the root filesystem, not the mirror, that gets remounted read-only, making the entire OS non-operative and requiring a hard reset. Upon reboot, the mirror is apparently attempted to be reconstructed but is now typically named /dev/md127.

Yes, I was attempting to create the mirror using two disks that had previously been partitioned with a GPT partition table and then formatted with one large ext4 filesystem. From everything I have read, this should be acceptable.

[NOTE: When mdadm says "/dev/sdd1 appears to contain an ext2fs file system", it is misidentifying the ext4fs -- probably due to a hard-coded error message that was never properly updated. As far as the partition types, GParted does not allow them to be formatted as type fd directly, but I do believe that mdadm tags them as such when it assembles them into the array.]

Based upon the comments below, this is what I tried:

1: I ran an extended S.M.A.R.T. surface test on all four 4TB drives (2 for mirror, 2 as future spares). Each test took over 8.5 hours and all disks reported without error. Exercising these disks individually has never caused a system panic.

2: Using GParted, I deleted the ext4 partitions from the sdc and sdd disks.

3: To make sure that the original GPT partition tables were eliminated I ran:

# sgdisk -Z /dev/sdc
# sgdisk -Z /dev/sdd

4: I recreated the array using the two unformatted disks.

# mdadm --create /dev/md2 --verbose --level=1 --metadata 1.2 --raid-devices=2 /dev/sdc /dev/sdd
mdadm: size set to 3906887360K
mdadm: array /dev/md2 started

5: I started monitoring the sync using "cat /proc/mdstat" and saw it advancing nicely.

After a couple of minutes the system panicked as usual and the root filesystem (sda1) was remounted RO and required a hard reset. Upon reboot the array was renamed /dev/md127 and in this case, it is in a "resync=PENDING" state and is not automatically attempting to sync. The intent was to create the GPT partition table and ext4 partition on the mirror once it finished syncing. (I know that I could have probably gone ahead and done this during the sync, but I am trying to isolate the steps in this process to see where the problem lies.)

Here is some new information that I found duplicated in the syslog and kern.log files. These messages were logged just prior to the remount-ro operation.

Nov 15 14:31:15 dymaxion kernel: [58171.002154] ata8.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Nov 15 14:31:15 dymaxion kernel: [58171.002163] ata8.00: failed command: IDENTIFY DEVICE
Nov 15 14:31:15 dymaxion kernel: [58171.002167] ata8.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 16 pio 512 in
Nov 15 14:31:15 dymaxion kernel: [58171.002167]          res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 15 14:31:15 dymaxion kernel: [58171.002169] ata8.00: status: { DRDY }
Nov 15 14:31:15 dymaxion kernel: [58171.002175] ata8: hard resetting link
Nov 15 14:31:15 dymaxion kernel: [58171.329795] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Nov 15 14:31:15 dymaxion kernel: [58171.330336] ata8.00: supports DRM functions and may not be fully accessible
Nov 15 14:31:15 dymaxion kernel: [58171.334346] ata8.00: disabling queued TRIM support
Nov 15 14:31:15 dymaxion kernel: [58171.339116] ata8.00: supports DRM functions and may not be fully accessible
Nov 15 14:31:15 dymaxion kernel: [58171.343149] ata8.00: disabling queued TRIM support
Nov 15 14:31:15 dymaxion kernel: [58171.347557] ata8.00: configured for UDMA/133
Nov 15 14:31:15 dymaxion kernel: [58171.347625] ata8: EH complete

This seems to indicate some sort of SATA error, although, at the moment I cannot interpret it.

So, does this provide any additional clues as to what may be wrong? I truly appreciate the help thus far. It got me thinking in a couple new directions. I hope someone can provide further insight or suggestions. Thanks.

---

Everything below here was added Saturday, December 20, 2014

---

As a final entry in this saga, I am providing the following information in the hopes that it will help others in the future.

I did manage to get in touch with US ASUS support regarding this problem. I received a replacement Z9PE-D8 WS motherboard which I installed and configured. When I ran my RAID tests I ended up observing exactly the same results as with the original motherboard. With the root filesystem drive attached to the Marvel controller:

• If the additional RAID 1 disks were on the Marvel controller, any attempt to perform a significant mdadm(8) operation on the array generated the kernel exception and errors noted above and the entire OS would panic.

• If the RAID disks were moved off of the Marvel controller, then mdadm(8) operations could be performed without problem and the system operated without problem.

Since I intended to mirror the root partition, I was anxious to see what would happen if the root filesystem was removed from the Marvel controller and the RAID was moved back onto it. Unfortunately, I could find no way to ever boot the OS if the root filesystem was moved to the on-board Intel C602 chipset. This was the case with both motherboards.

[NOTE: If anyone has a clue why this could not be done, I would appreciate hearing the reason. For example, does GRUB2 store some particular information at installation time that is controller-specific?]

Therefore, I bit the bullet and decided to completely reinstall the latest Ubuntu Server version 14.10 and mirror the root filesystem as part of the install process. I moved the SSDs to the pair of SATA-III ports controlled by the Intel controller and performed a fresh install. Everything worked fine.

Now, with a running system with a mirrored root, I attached the two 4TB drives to the Marvel controller and tried to construct a new RAID 1 array. The array soon failed. Thus, we can conclusively conclude that the Marvel controller is doing something that is incompatible with software RAID management.

I moved the 4TB drives to SATA-II ports controlled by the Intel C602 and everything worked and continues to work without a hitch. ASUS engineering is looking into the problem while I am left with a machine where four of the original six SATA-III ports are unusable.

The lesson is that anyone considering a Linux machine that uses Marvel PCIe 9230 controller should be concerned about RAID compatibility.

I hope this information is useful. If anyone else discovers similar problems with the Marvel controller and can shed further light on the subject, please contact me. Thanks. ~

Answer 1

The biggest problem I see is this

mdadm: /dev/sdd1 appears to contain an ext2fs file system

Also, those partitions should be marked as RAID members (type fd), not Linux filesystems.

Which means there are superblocks that extfs tools can latch onto, like fsck, and mess up your world bad. I would strongly recommend that you completely wipe the drives before adding them to them array using dd like so.

dd if=/dev/zero of=/dev/bye-bye-entire-sd-device

Make sure you're formatting the MD device with your filesystem, not the members.

If all that works out and you're still seeing random corruption then you likely have some marginal memory that's writing back garbage every once in a while and destroying your disks.

For further reference: https://raid.wiki.kernel.org/index.php/RAID_setup

Answer 2

Since you didn't mention creating a filesystem on the raid array and mounting it after creating the array, and mdadm warned you that /dev/sdc1 already has an ext2 filesystem in it, I'm guessing you mean you already have a filesystem in /dev/sdc1, and that is what is being remounted read only. This is because creating a raid array out of a disk or partition is generally a destructive operation, hence why mdadm warned you. By writing the raid metadata to the partition, you have damaged the existing filesystem there.

At this point you need to try and undo the damage you have done if you want to recover the existing data in /dev/sdc1. Start by unmounting the old filesystem, then blowing away the raid superblocks you created, and then fsck the old filesystem and hope it can be repaired:

umount /dev/sdc1
mdadm --zero-superblocks /dev/sdc1 /dev/sdd1
e2fsck -fy /dev/sdc1

To upgrade an existing filesystem to a raid1, you first need to create the raid array using only the new disk, then manually copy all of your files from the old FS to the new one:

mdadm --create --level 1 -n 2 /dev/sdd1 missing
mkfs.ext4 /dev/md0
mkdir /mnt/new
mkdir /mnt/old
mount /dev/md0 /mnt/new
mount /dev/sdc1 /mnt/old
cp -ax /mnt/old/* /mnt/new/
umount /mnt/old
umount /mnt/new
rmdir /mnt/old
rmdir /mnt/new

Now edit your /etc/fstab to mount the new volume in /dev/md0 instead of the old one in /dev/sdc1, and finally you can hand /dev/sdc1 over to md to mirror everything in /dev/sdd1 onto:

mdadm /dev/md0 --add /dev/sdc1

You can use blkid to look up the UUID of the new filesystem in the raid array and use that to replace the old UUID in /etc/fstab. Also all of these commands must be run as root, so you will want to sudo -s first to become root.

Finally, FYI, you might want to use raid10 instead of raid1. With the offset layout ( -p o2 to mdadm ) and a largish chunk size size ( -c 1024 to 4096 ), you can get the redundancy of raid1 plus the sequential read throughput of raid0.

Answer 3

Looking for love in all the wrong places ....

Thank you psusi and ppetraki for your helpful replies. You each gave me additional insights into how RAID functions under Linux.

It turns out that there was nothing wrong with the disks or the mdadm commands that I was using to create and manipulate the RAID arrays. Once I discovered the ata8 kernel messages, I searched the internet using them as a key and found others reporting similar messages which were associated with a Marvel SATA controller. I have an ASUS Z9PE-D8 WS motherboard with an on board Marvel PCIe 9230 controller driving four SATA-III ports which were being used for these disks. I unplugged the drives from these ports, connected them to other SATA ports on the board that were being driven by an Intel C602 chipset and rebooted. At this point I was able to build multiple arrays, reconfigure them, etc. without any problem!

The single SSD with the root filesystem is still attached to the Marvel controller and has exhibited no problem running. However, I now have no plans to attempt to mirror this drive until it is also removed from the Marvel controller.

I am attempting to get some information out of ASUS regarding this issue. I do not know it it could indicate a hardware or a BIOS problem. So far, ASUS technical support has been slow to respond to my requests. I'm not impressed with their service.

If anyone had additional information related to the Marvel controller problems, I would certainly appreciate hearing about it.

So, I am back in business for the time being, four SATA-III ports shy of a properly working system. Thanks again for the assistance.