My OS is Ubuntu 12.04.2 LTS with this kernel: 3.2.0-49-generic #75-Ubuntu SMP Tue Jun 18 17:39:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
I have a raid 5 array composed of 3 hard drives and suddenly, when it started being inactive at boot time. Since the home directory is mounted on it, the system cannot boot and it's asking for manual user intervention. I found similar reports in forums, but most of them happen to have a defective hard drive which is not the case for me.
Stopping the array (mdadm --stop /dev/md0) and starting it again (mdadm --assemble --scan /dev/md0) shows no errors (there is no complaining or array rebuilding) and then it can be mounted properly (manual mount) so why it cannot be brought up at boot?
After checking the smartctl for all the hard drives composing the raid array (sda, sdb, sdc), I could not observe any error (no Current_Pending_Sector, UDMA_CRC_Error_Count, Offline_Uncorrectable). Short and long tests have been ran already.
One thing I noticed which is mostly the cause of the problem is that grub-probe returns this error: "error: found two disks with the index 1 for RAID md/0."
Running the same command with -v (verbose output), I can spot two lines commenting "grub-probe: info: Found array md/0 (mdraid1x)." right after probing hd0 and hd1 which are mapped on sda and sdb correspondingly. So sdc doesn't have readable by grub metadata for the raid? People facing this problem suggested to update the raid metadata from 0.90 to 1.x but my raid is already using 1.2.
I tried to manually make sdc hard drive to fail twice (the first time just removed it and re-added it and the second time by using mdadm --zero-superblock /dev/sdc) and forced the raid to rebuild, but the error cannot go away so now I am stuck. Does anyone have a clue on what the problem might be and how can it be fixed?
Below there is a list of commands and their output that I used to diagnose the problem:
/proc/stat after boot
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdc1[3](S) sda1[4](S) sdb[5](S)
5860540617 blocks super 1.2
unused devices: <none>
/etc/mdadm/mdadm.conf
# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e name=ubuntu:0
# This file was auto-generated on Mon, 27 Aug 2012 17:33:16 +0300
# by mkconf $Id$
mdadm --examine --scan
# mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e name=ubuntu:0
mdadm --detail
# mdadm --detail --scan
mdadm: cannot open /dev/md/0: No such file or directory
# mdadm --detail --scan /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
mdadm --stop /dev/md0 && mdadm --assemble --scan /dev/md0 && mdadm --detail /dev/md0
# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble --scan /dev/md0
mdadm: /dev/md0 has been started with 3 drives.
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda1[4] sdc1[3] sdb1[5]
3907025920 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Mar 24 15:31:43 2012
Raid Level : raid5
Array Size : 3907025920 (3726.03 GiB 4000.79 GB)
Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Sun Jul 21 22:53:21 2013
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : ubuntu:0
UUID : 1b273efc:62f3bc36:4579f11d:15bbc75e
Events : 319386
Number Major Minor RaidDevice State
4 8 1 0 active sync /dev/sda1
5 8 17 1 active sync /dev/sdb1
3 8 33 2 active sync /dev/sdc1
mdadm --detail --scan && mdadm --examine --scan
# mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=ubuntu:0 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e
# mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e name=ubuntu:0
grub-probe -v /
# grub-probe -v /
grub-probe: info: cannot open `/boot/grub/device.map'.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: scanning hd0 for LVM.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: scanning hd1 for LVM.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: scanning hd2 for LVM.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: scanning hd3 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Found array md/0 (mdraid1x).
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd0,msdos1.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd1,msdos1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd2,msdos1.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3,msdos2.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3,msdos1.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd0,msdos1.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Found array md/0 (mdraid1x).
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd1,msdos1.
grub-probe: info: the size of hd1 is 3907029168.
error: found two disks with the index 1 for RAID md/0.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd2,msdos1.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3,msdos2.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3,msdos1.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: scanning md/0 for LVM.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd0 for LVM.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: scanning hd0,msdos1 for LVM.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd1 for LVM.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: scanning hd1,msdos1 for LVM.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd2 for LVM.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: scanning hd2,msdos1 for LVM.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd3 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: scanning hd3,msdos2 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd3,msdos1 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: /dev/sdd1 starts from 2048.
grub-probe: info: opening the device hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Partition 0 starts from 2048.
grub-probe: info: opening hd3,msdos1.
grub-probe: info: the size of hd3 is 250069680.
ext2
/boot/grub/device.map
# cat /boot/grub/device.map
(hd0) /dev/sda
(hd1) /dev/sdb
(hd2) /dev/sdc
(hd3) /dev/sdd