2

It's rare I ask questions, but Google has turned up a blank on this one, so here I am.

I have a problem with my mdadm software Raid running on Ubuntu 14.04 Server edition. Specifically, I seem to have 2 Raid arrays set with the same UUID, one being an old one, one being a new one, which means that it can't be assembled at boot, causing boot to hang as it is unable to mount some of my file systems.

I will detail what I did, and where I think this issue has presented itself. What I am looking for is how to scrub the details of the now obsolete array so that my Raid assembles correctly at boot.

Way back when I had a Raid1 Array consistent of two 2TB drives. The system does not sit on this array, it has a dedicated disk, so this is purely for data.

I started running out of space, so migrated these 2 drives to Raid5. So far, so good.

I then expanded my array by adding 2 2TB spares and growing the array into these spares. This is where I think the issue has occurred.

I then decided, given the size of the Array and rebuild time, I would prefer to run Raid6, so I striped 2 existing 1TB drives to make a 2TB Raid0 and grew Raid 6 over these drives. This is where I am now at, a Raid 6 consisting of 4 X 2TB drives + 1 X 2TB Raid 0 (10 TB total, 6 TB usable space, 4 TB parity).

At some point during this upgrade it looks like I screwed something up. The results of examining the Raid shows the following:

sudo mdadm --examine --scan -v
ARRAY /dev/md/1 level=raid0 metadata=1.2 num-devices=2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
   devices=/dev/sdd1,/dev/sda1
ARRAY /dev/md/0 level=raid5 metadata=1.2 num-devices=2 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
   spares=2   devices=/dev/sde,/dev/sdf
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=5 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
   devices=/dev/sdg1,/dev/sdh1,/dev/sde1,/dev/sdf1,/dev/md/1

See those 2 /dev/md0 with different Raid levels, but the same UUID? Sucks, yeah? What it looks like is that the Raid5, with 2 spares has remained in config and has not been overwritten by growing the same array into those 2 spares and subsequently into Raid6. Crucially it looks like I may have screwed up by having my initial array referencing the whole disks (/dev/sd[ef]) and my Raid6 referencing the whole disk partitions (/dev/sd[efgh]1).

The upshot of this is that I cannot assemble the array using:

sudo mdadm --assemble --scan

as it tries to assemble the Raid5 with 2 spares and cannot complete. Instead I have to issue the following to assemble the array:

sudo mdadm --assemble /dev/md0 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/md1

Which works fine and my Raid6 becomes available again at /dev/md0... HOWEVER, it means that my server now requires manual intervention when it is unable to mount the filesystems to the Raid at boot, which is incredibly inconvenient, as it means I have to physically go to the server to [S]kip the failed mount.

Below are the outputs of my conf file and other mdadm outputs (whilst my Raid6 is successfully running) to help further diagnose a solution.

mdstat:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdg1[0] md1[5] sdf1[3] sde1[4] sdh1[2]
      5860145664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]

md1 : active raid0 sdd1[1] sda1[0]
      1953522688 blocks super 1.2 512k chunks

unused devices: <none>

mdadm --detail

sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Sun Sep 27 14:00:17 2015
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : team-canfield:0  (local to host team-canfield)
           UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
         Events : 1231852

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       2       8      113        1      active sync   /dev/sdh1
       4       8       65        2      active sync   /dev/sde1
       3       8       81        3      active sync   /dev/sdf1
       5       9        1        4      active sync   /dev/md/1

mdadm.conf:

cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/1 metadata=1.2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
ARRAY /dev/md/0 metadata=1.2 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0

mdadm --examine --scan -vvvv:

sudo mdadm --examine --scan -vvvv
mdadm: No md superblock detected on /dev/md0.
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
   Raid Devices : 5

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e39fa659:9010941e:034e58b6:f88f7542

    Update Time : Sun Sep 27 14:00:17 2015
       Checksum : 18244273 - correct
         Events : 1231852

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdg:
   MBR Magic : aa55
Partition[0] :   3907027120 sectors at         2048 (type fd)
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
   Raid Devices : 5

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : cc416705:bf6220eb:9201a515:c044aabd

    Update Time : Sun Sep 27 14:00:17 2015
       Checksum : 6af29ee4 - correct
         Events : 1231852

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdh:
   MBR Magic : aa55
Partition[0] :   3907027120 sectors at         2048 (type fd)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
   Raid Devices : 5

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : dbd3c13e:269dde52:0b07e3cf:d8d2e4fc

    Update Time : Sun Sep 27 14:00:17 2015
       Checksum : 583feee - correct
         Events : 1231852

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid5
   Raid Devices : 2

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 1953381888 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 8089c55e:df3a826b:7df2e4b9:dbff7a97

    Update Time : Sat Sep  5 14:59:00 2015
       Checksum : 7f43d8a - correct
         Events : 514

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AA ('A' == active, '.' == missing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
   Raid Devices : 5

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : de9ae300:e4382032:a5fd89c9:c966255b

    Update Time : Sun Sep 27 14:00:17 2015
       Checksum : feceec37 - correct
         Events : 1231852

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid5
   Raid Devices : 2

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 1953381888 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 3b86b670:f4812448:e474f7b5:b214454c

    Update Time : Sat Sep  5 14:49:56 2015
       Checksum : a7651675 - correct
         Events : 513

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AA ('A' == active, '.' == missing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 913834a2:63afa2cd:a84e864a:96353af5
           Name : fileserver:1
  Creation Time : Thu Sep 10 11:52:39 2015
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 1953523104 (931.51 GiB 1000.20 GB)
    Data Offset : 16 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 8745a084:7e031e4f:da72b198:e548d755

    Update Time : Thu Sep 10 11:52:39 2015
       Checksum : 48772d61 - correct
         Events : 0

     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   1953523120 sectors at         2048 (type 83)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdc:
   MBR Magic : aa55
Partition[0] :    976771072 sectors at         2048 (type 83)
mdadm: No md superblock detected on /dev/dm-2.
mdadm: No md superblock detected on /dev/dm-1.
mdadm: No md superblock detected on /dev/dm-0.
mdadm: No md superblock detected on /dev/sdb5.
/dev/sdb2:
   MBR Magic : aa55
Partition[0] :   1953021952 sectors at            2 (type 8e)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdb:
   MBR Magic : aa55
Partition[0] :       497664 sectors at         2048 (type 83)
Partition[1] :   1953021954 sectors at       501758 (type 05)
/dev/md/1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
           Name : team-canfield:0  (local to host team-canfield)
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
   Raid Devices : 5

 Avail Dev Size : 3906783232 (1862.90 GiB 2000.27 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : cc54a0b1:0a5cf703:e7e0b1d2:0bab9231

    Update Time : Sun Sep 27 14:00:17 2015
       Checksum : 60f83822 - correct
         Events : 1231852

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 913834a2:63afa2cd:a84e864a:96353af5
           Name : fileserver:1
  Creation Time : Thu Sep 10 11:52:39 2015
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 1953523104 (931.51 GiB 1000.20 GB)
    Data Offset : 16 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 8e41dd73:0e90d173:852facd8:f3009a34

    Update Time : Thu Sep 10 11:52:39 2015
       Checksum : 7b252ab0 - correct
         Events : 0

     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sda:
   MBR Magic : aa55
Partition[0] :   1953523120 sectors at         2048 (type 83)

This last one shows disks /dev/sd[ef] both being part of a Raid5 whilst their primary (and only) partition is part of the Raid6

So, is there a way, without losing my data, I can destroy the Raid5 information? I do not have the backup space to rebuild my array and copy everything back on.

I would imagine I could update mdadm.conf to the following:

DEVICE partitions

and it would ignore the Raid5, however, I feel this just papering over the cracks, and I will undoubtedly hit similar issues again next time I change my Raid configuration. Really, I just want to destroy all evidence of this phantom Raid5 with the same UUID as my active Raid6. Any takers?

Many thanks!

1 Answer 1

0

Ok, I think I have fixed it. I will go through the steps I took in case anyone else has this issue. Also, this technique is risky, as I am currently running a degraded Raid6 Array from 3 partions of 5.

Stop the Array:

$ sudo mdadm --stop /dev/md0
 mdadm: stopped /dev/md0

Zero the superblock from the first disk in the obsolete Raid5 array

$ sudo mdadm --zero-superblock /dev/sdf

Examine the arrays to see if it removed it:

$ sudo mdadm --examine --scan -v
ARRAY /dev/md/1 level=raid0 metadata=1.2 num-devices=2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
   devices=/dev/sdd1,/dev/sda1
ARRAY /dev/md/0 level=raid5 metadata=1.2 num-devices=2 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
   spares=1   devices=/dev/sde
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=5 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
   devices=/dev/sdg1,/dev/sdh1,/dev/md/1

Strangely I lost both of the partitions (sd[ef]1) from my Raid6 at this point. But the good thing was the Raid5 was just going with one spare.

Zero the remaining superblock:

$ sudo mdadm --zero-superblock /dev/sde

Check again:

$ sudo mdadm --examine --scan -v
ARRAY /dev/md/1 level=raid0 metadata=1.2 num-devices=2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
   devices=/dev/sdd1,/dev/sda1
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=5 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
   devices=/dev/sdg1,/dev/sdh1,/dev/md/1

Obsolete Raid5 has now disappeared. Now the moment of truth, is it possible to recover the partitions back into the Raid6:

$ sudo mdadm --assemble /dev/md0 /dev/sdg1 /dev/sdh1 /dev/md1 /dev/sde1 /dev/sdf1 --force
mdadm: cannot open device /dev/sde1: No such file or directory
mdadm: /dev/sde1 has no superblock - assembly aborted

$ sudo mdadm --assemble /dev/md0 /dev/sdg1 /dev/sdh1 /dev/md1 /dev/sdf1 --force
mdadm: cannot open device /dev/sdf1: No such file or directory
mdadm: /dev/sdf1 has no superblock - assembly aborted

No it's not :/ This is bad news and I fear the worst for my data. Try to assemble the 3 remaining partitions:

$ sudo mdadm --assemble /dev/md0 /dev/sdg1 /dev/sdh1 /dev/md1  --force
mdadm: /dev/md0 has been started with 3 drives (out of 5).

Phew! Success!

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdg1[0] md1[5] sdh1[2]
      5860145664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [UU__U]

md1 : active raid0 sdd1[1] sda1[0]
      1953522688 blocks super 1.2 512k chunks

Ok, we're seriously degraded, but at least it's built. I mount the array to make sure my data is intact and breath a huge sigh of relief when I see it's all still there. NB: It may be wise not to mount the array at this point. I think this may be the difference between being able to issues a --re-add rather than an --add later in these steps.

I feel it would be a good idea to reboot, so I do. I still get a hang on boot, but that is unsurprising as my array is degraded.

When it comes back online, I examine again:

$ sudo mdadm --examine --scan -v
ARRAY /dev/md/1 level=raid0 metadata=1.2 num-devices=2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
   devices=/dev/sdd1,/dev/sda1
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=5 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
   devices=/dev/sdg1,/dev/sdh1,/dev/sdf1,/dev/sde1,/dev/md/1

Could this be? It now sees all 5 partitions of the Raid6 and the Raid5 is nowhere to be seen.

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdg1[0](S) sdh1[2](S) sde1[4](S) sdf1[3](S)
      7813529952 blocks super 1.2

md1 : active raid0 sdd1[1] sda1[0]
      1953522688 blocks super 1.2 512k chunks

unused devices: <none>

Oh dear, of course this was too good to be true. I'm missing a partition from the array and all the present members are listed as spares.

Well, we've got this far. Let's stop the array:

$ sudo mdadm --stop /dev/md0
mdadm: stopped /dev/md0

And attempt to assemble scan:

$ sudo mdadm --assemble --scan

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdg1[0] md1[5] sdh1[2]
      5860145664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [UU__U]

md1 : active raid0 sdd1[1] sda1[0]
      1953522688 blocks super 1.2 512k chunks

Amazing! We're back online, and it's the first time I've been able to issue this command successfully since I upgraded from Raid5 to Raid6. We are, however, still degraded with /dev/sd[ef]1 nowhere to be seen.

I'll stop the array and attempt a force re-assemble explicitly naming the member partitions:

$ sudo mdadm --stop /dev/md0
mdadm: stopped /dev/md0

$ sudo mdadm --assemble /dev/md0 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/md1 -v --force
mdadm: looking for devices for /dev/md0
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/md1 is identified as a member of /dev/md0, slot 4.
mdadm: added /dev/sdh1 to /dev/md0 as 1
mdadm: added /dev/sde1 to /dev/md0 as 2 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md0 as 3 (possibly out of date)
mdadm: added /dev/md1 to /dev/md0 as 4
mdadm: added /dev/sdg1 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 3 drives (out of 5).

Ok, well this is very good news. Apparently the superblock still exists on these partitions, even though the 2 partitions can't actually be added in an active state at the moment.

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdg1[0] md1[5] sdh1[2]
      5860145664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [UU__U]

md1 : active raid0 sdd1[1] sda1[0]
      1953522688 blocks super 1.2 512k chunks

unused devices: <none>

$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Sep  4 00:57:56 2015
     Raid Level : raid6
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
   Raid Devices : 5
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Thu Oct  1 23:57:14 2015
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : team-canfield:0  (local to host team-canfield)
           UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
         Events : 1231860

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       2       8      113        1      active sync   /dev/sdh1
       2       0        0        2      removed
       3       0        0        3      removed
       5       9        1        4      active sync   /dev/md/1

So they've gone back in a removed state. Let's try to re-add them:

$ sudo mdadm --manage /dev/md0 --re-add /dev/sd[ef]1
mdadm: --re-add for /dev/sde1 to /dev/md0 is not possible

$ sudo mdadm --manage /dev/md0 --re-add /dev/sdf1
mdadm: --re-add for /dev/sdf1 to /dev/md0 is not possible

Damn, of course not. Well, let's examine the events on the partitions to see if they are not truly recognised as part of the array:

$ sudo mdadm --examine /dev/sd[a-z]1 | egrep 'Event|/dev/sd'
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.
/dev/sda1:
         Events : 0
/dev/sdd1:
         Events : 0
/dev/sde1:
         Events : 1231852
/dev/sdf1:
         Events : 1231852
/dev/sdg1:
         Events : 1231860
/dev/sdh1:
         Events : 1231860

Ok, apparently just 8 events from 1.2m is enough to make a --re-add impossible. This is a shame, and is probably from me mounting the raid to check it was still intact earlier in the process, otherwise it looks like I had actually managed to zero the superblocks from the outlying disks whilst leaving the superblocks and data on the partitions intact (I obviously don't actually know how superblocks work).

Well, at this point there is no real choice other than to add the two partitions back into the array and let it rebuild:

$ sudo mdadm --manage /dev/md0 --add /dev/sd[ef]1
mdadm: added /dev/sde1
mdadm: added /dev/sdf1

Excellent! Now let's check progress:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdf1[7] sde1[6] sdg1[0] md1[5] sdh1[2]
      5860145664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [UU__U]
      [>....................]  recovery =  0.0% (84312/1953381888) finish=1930.4min speed=16862K/sec

md1 : active raid0 sdd1[1] sda1[0]
      1953522688 blocks super 1.2 512k chunks

A day to build isn't too bad, seeing as it took 7 days to migrate from Raid5 to Raid6. This is going to be a scary 24hours as I pray nothing happens to my array during the rebuild process, but it looks like it's a success.

So we learned:

  • Raid6 is, as expected, perfectly functional when it loses 2 partitions from 5
  • Don't mix your drives and partitions when using mdadm.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .