It's rare I ask questions, but Google has turned up a blank on this one, so here I am.
I have a problem with my mdadm software Raid running on Ubuntu 14.04 Server edition. Specifically, I seem to have 2 Raid arrays set with the same UUID, one being an old one, one being a new one, which means that it can't be assembled at boot, causing boot to hang as it is unable to mount some of my file systems.
I will detail what I did, and where I think this issue has presented itself. What I am looking for is how to scrub the details of the now obsolete array so that my Raid assembles correctly at boot.
Way back when I had a Raid1 Array consistent of two 2TB drives. The system does not sit on this array, it has a dedicated disk, so this is purely for data.
I started running out of space, so migrated these 2 drives to Raid5. So far, so good.
I then expanded my array by adding 2 2TB spares and growing the array into these spares. This is where I think the issue has occurred.
I then decided, given the size of the Array and rebuild time, I would prefer to run Raid6, so I striped 2 existing 1TB drives to make a 2TB Raid0 and grew Raid 6 over these drives. This is where I am now at, a Raid 6 consisting of 4 X 2TB drives + 1 X 2TB Raid 0 (10 TB total, 6 TB usable space, 4 TB parity).
At some point during this upgrade it looks like I screwed something up. The results of examining the Raid shows the following:
sudo mdadm --examine --scan -v
ARRAY /dev/md/1 level=raid0 metadata=1.2 num-devices=2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
devices=/dev/sdd1,/dev/sda1
ARRAY /dev/md/0 level=raid5 metadata=1.2 num-devices=2 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
spares=2 devices=/dev/sde,/dev/sdf
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=5 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
devices=/dev/sdg1,/dev/sdh1,/dev/sde1,/dev/sdf1,/dev/md/1
See those 2 /dev/md0 with different Raid levels, but the same UUID? Sucks, yeah? What it looks like is that the Raid5, with 2 spares has remained in config and has not been overwritten by growing the same array into those 2 spares and subsequently into Raid6. Crucially it looks like I may have screwed up by having my initial array referencing the whole disks (/dev/sd[ef]) and my Raid6 referencing the whole disk partitions (/dev/sd[efgh]1).
The upshot of this is that I cannot assemble the array using:
sudo mdadm --assemble --scan
as it tries to assemble the Raid5 with 2 spares and cannot complete. Instead I have to issue the following to assemble the array:
sudo mdadm --assemble /dev/md0 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/md1
Which works fine and my Raid6 becomes available again at /dev/md0... HOWEVER, it means that my server now requires manual intervention when it is unable to mount the filesystems to the Raid at boot, which is incredibly inconvenient, as it means I have to physically go to the server to [S]kip the failed mount.
Below are the outputs of my conf file and other mdadm outputs (whilst my Raid6 is successfully running) to help further diagnose a solution.
mdstat:
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdg1[0] md1[5] sdf1[3] sde1[4] sdh1[2]
5860145664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
md1 : active raid0 sdd1[1] sda1[0]
1953522688 blocks super 1.2 512k chunks
unused devices: <none>
mdadm --detail
sudo mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid6
Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Sun Sep 27 14:00:17 2015
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : team-canfield:0 (local to host team-canfield)
UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Events : 1231852
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
2 8 113 1 active sync /dev/sdh1
4 8 65 2 active sync /dev/sde1
3 8 81 3 active sync /dev/sdf1
5 9 1 4 active sync /dev/md/1
mdadm.conf:
cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/1 metadata=1.2 UUID=913834a2:63afa2cd:a84e864a:96353af5 name=fileserver:1
ARRAY /dev/md/0 metadata=1.2 UUID=18235f5c:bd4892c0:7616a677:626ae2e8 name=team-canfield:0
mdadm --examine --scan -vvvv:
sudo mdadm --examine --scan -vvvv
mdadm: No md superblock detected on /dev/md0.
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e39fa659:9010941e:034e58b6:f88f7542
Update Time : Sun Sep 27 14:00:17 2015
Checksum : 18244273 - correct
Events : 1231852
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdg:
MBR Magic : aa55
Partition[0] : 3907027120 sectors at 2048 (type fd)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : cc416705:bf6220eb:9201a515:c044aabd
Update Time : Sun Sep 27 14:00:17 2015
Checksum : 6af29ee4 - correct
Events : 1231852
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdh:
MBR Magic : aa55
Partition[0] : 3907027120 sectors at 2048 (type fd)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : dbd3c13e:269dde52:0b07e3cf:d8d2e4fc
Update Time : Sun Sep 27 14:00:17 2015
Checksum : 583feee - correct
Events : 1231852
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid5
Raid Devices : 2
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 1953381888 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8089c55e:df3a826b:7df2e4b9:dbff7a97
Update Time : Sat Sep 5 14:59:00 2015
Checksum : 7f43d8a - correct
Events : 514
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AA ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : de9ae300:e4382032:a5fd89c9:c966255b
Update Time : Sun Sep 27 14:00:17 2015
Checksum : feceec37 - correct
Events : 1231852
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid5
Raid Devices : 2
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 1953381888 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3b86b670:f4812448:e474f7b5:b214454c
Update Time : Sat Sep 5 14:49:56 2015
Checksum : a7651675 - correct
Events : 513
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AA ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 913834a2:63afa2cd:a84e864a:96353af5
Name : fileserver:1
Creation Time : Thu Sep 10 11:52:39 2015
Raid Level : raid0
Raid Devices : 2
Avail Dev Size : 1953523104 (931.51 GiB 1000.20 GB)
Data Offset : 16 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8745a084:7e031e4f:da72b198:e548d755
Update Time : Thu Sep 10 11:52:39 2015
Checksum : 48772d61 - correct
Events : 0
Chunk Size : 512K
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 1953523120 sectors at 2048 (type 83)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdc:
MBR Magic : aa55
Partition[0] : 976771072 sectors at 2048 (type 83)
mdadm: No md superblock detected on /dev/dm-2.
mdadm: No md superblock detected on /dev/dm-1.
mdadm: No md superblock detected on /dev/dm-0.
mdadm: No md superblock detected on /dev/sdb5.
/dev/sdb2:
MBR Magic : aa55
Partition[0] : 1953021952 sectors at 2 (type 8e)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdb:
MBR Magic : aa55
Partition[0] : 497664 sectors at 2048 (type 83)
Partition[1] : 1953021954 sectors at 501758 (type 05)
/dev/md/1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 18235f5c:bd4892c0:7616a677:626ae2e8
Name : team-canfield:0 (local to host team-canfield)
Creation Time : Fri Sep 4 00:57:56 2015
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 3906783232 (1862.90 GiB 2000.27 GB)
Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : cc54a0b1:0a5cf703:e7e0b1d2:0bab9231
Update Time : Sun Sep 27 14:00:17 2015
Checksum : 60f83822 - correct
Events : 1231852
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 913834a2:63afa2cd:a84e864a:96353af5
Name : fileserver:1
Creation Time : Thu Sep 10 11:52:39 2015
Raid Level : raid0
Raid Devices : 2
Avail Dev Size : 1953523104 (931.51 GiB 1000.20 GB)
Data Offset : 16 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8e41dd73:0e90d173:852facd8:f3009a34
Update Time : Thu Sep 10 11:52:39 2015
Checksum : 7b252ab0 - correct
Events : 0
Chunk Size : 512K
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing)
/dev/sda:
MBR Magic : aa55
Partition[0] : 1953523120 sectors at 2048 (type 83)
This last one shows disks /dev/sd[ef] both being part of a Raid5 whilst their primary (and only) partition is part of the Raid6
So, is there a way, without losing my data, I can destroy the Raid5 information? I do not have the backup space to rebuild my array and copy everything back on.
I would imagine I could update mdadm.conf to the following:
DEVICE partitions
and it would ignore the Raid5, however, I feel this just papering over the cracks, and I will undoubtedly hit similar issues again next time I change my Raid configuration. Really, I just want to destroy all evidence of this phantom Raid5 with the same UUID as my active Raid6. Any takers?
Many thanks!