RAID 5 Inactive at boot - error: found two disks with the index 1 for RAID md/0

Question

My OS is Ubuntu 12.04.2 LTS with this kernel: 3.2.0-49-generic #75-Ubuntu SMP Tue Jun 18 17:39:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I have a raid 5 array composed of 3 hard drives and suddenly, when it started being inactive at boot time. Since the home directory is mounted on it, the system cannot boot and it's asking for manual user intervention. I found similar reports in forums, but most of them happen to have a defective hard drive which is not the case for me.

Stopping the array (mdadm --stop /dev/md0) and starting it again (mdadm --assemble --scan /dev/md0) shows no errors (there is no complaining or array rebuilding) and then it can be mounted properly (manual mount) so why it cannot be brought up at boot?

After checking the smartctl for all the hard drives composing the raid array (sda, sdb, sdc), I could not observe any error (no Current_Pending_Sector, UDMA_CRC_Error_Count, Offline_Uncorrectable). Short and long tests have been ran already.

One thing I noticed which is mostly the cause of the problem is that grub-probe returns this error: "error: found two disks with the index 1 for RAID md/0."

Running the same command with -v (verbose output), I can spot two lines commenting "grub-probe: info: Found array md/0 (mdraid1x)." right after probing hd0 and hd1 which are mapped on sda and sdb correspondingly. So sdc doesn't have readable by grub metadata for the raid? People facing this problem suggested to update the raid metadata from 0.90 to 1.x but my raid is already using 1.2.

I tried to manually make sdc hard drive to fail twice (the first time just removed it and re-added it and the second time by using mdadm --zero-superblock /dev/sdc) and forced the raid to rebuild, but the error cannot go away so now I am stuck. Does anyone have a clue on what the problem might be and how can it be fixed?

Below there is a list of commands and their output that I used to diagnose the problem:

/proc/stat after boot

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : inactive sdc1[3](S) sda1[4](S) sdb[5](S)
      5860540617 blocks super 1.2

unused devices: <none>

/etc/mdadm/mdadm.conf

# cat /etc/mdadm/mdadm.conf 
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e name=ubuntu:0

# This file was auto-generated on Mon, 27 Aug 2012 17:33:16 +0300
# by mkconf $Id$

mdadm --examine --scan

# mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e name=ubuntu:0

mdadm --detail

# mdadm --detail --scan
mdadm: cannot open /dev/md/0: No such file or directory

# mdadm --detail --scan /dev/md0 
mdadm: md device /dev/md0 does not appear to be active.

mdadm --stop /dev/md0 && mdadm --assemble --scan /dev/md0 && mdadm --detail /dev/md0

# mdadm --stop /dev/md0 
mdadm: stopped /dev/md0

# mdadm --assemble --scan /dev/md0
mdadm: /dev/md0 has been started with 3 drives.

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sda1[4] sdc1[3] sdb1[5]
      3907025920 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

# mdadm --detail /dev/md0 
/dev/md0:
        Version : 1.2
  Creation Time : Sat Mar 24 15:31:43 2012
     Raid Level : raid5
     Array Size : 3907025920 (3726.03 GiB 4000.79 GB)
  Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun Jul 21 22:53:21 2013
          State : clean 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ubuntu:0
           UUID : 1b273efc:62f3bc36:4579f11d:15bbc75e
         Events : 319386

    Number   Major   Minor   RaidDevice State
       4       8        1        0      active sync   /dev/sda1
       5       8       17        1      active sync   /dev/sdb1
       3       8       33        2      active sync   /dev/sdc1

mdadm --detail --scan && mdadm --examine --scan

# mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=ubuntu:0 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e

# mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=1b273efc:62f3bc36:4579f11d:15bbc75e name=ubuntu:0

grub-probe -v /

# grub-probe -v /
grub-probe: info: cannot open `/boot/grub/device.map'.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for dmraid_nv RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: scanning hd0 for LVM.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: scanning hd1 for LVM.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: scanning hd2 for LVM.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: scanning hd3 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Found array md/0 (mdraid1x).
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd0,msdos1.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd1,msdos1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd2,msdos1.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3,msdos2.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid09 RAID devices on disk hd3,msdos1.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd0.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd0,msdos1.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: Found array md/0 (mdraid1x).
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd1.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd1,msdos1.
grub-probe: info: the size of hd1 is 3907029168.
error: found two disks with the index 1 for RAID md/0.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd2.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd2,msdos1.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3,msdos2.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Scanning for mdraid1x RAID devices on disk hd3,msdos1.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: scanning md/0 for LVM.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd0 for LVM.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: scanning hd0,msdos1 for LVM.
grub-probe: info: the size of hd0 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd1 for LVM.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: scanning hd1,msdos1 for LVM.
grub-probe: info: the size of hd1 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd2 for LVM.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: scanning hd2,msdos1 for LVM.
grub-probe: info: the size of hd2 is 3907029168.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd3 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: scanning hd3,msdos2 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: scanning hd3,msdos1 for LVM.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: no LVM signature found.
grub-probe: info: /dev/sdd1 starts from 2048.
grub-probe: info: opening the device hd3.
grub-probe: info: the size of hd3 is 250069680.
grub-probe: info: Partition 0 starts from 2048.
grub-probe: info: opening hd3,msdos1.
grub-probe: info: the size of hd3 is 250069680.
ext2

/boot/grub/device.map

# cat /boot/grub/device.map
(hd0)   /dev/sda
(hd1)   /dev/sdb
(hd2)   /dev/sdc
(hd3)   /dev/sdd

Answer 1

I solved the problem by removing each disk of the raid (one by one), zeroing the superblock and the MBR, added them back to the raid and waited for the rebuild.

After I did that for /dev/sdb, the problem was solved and now the grub-probe shows only one line with "grub-probe: info: Found array md/0 (mdraid1x)." instead of two as it happened before (look at the question).

So it must be the other way around than what I thought in the beginning about the index error. My thought was that this index should be present in every disk being part of the raid, that's why I was erasing sdc that grub-probe was not showing any "grub-probe: info: Found array md/0 (mdraid1x)." message.

Eventually it looks like only one of them must have it and if it is in more than one hard disk this error "error: found two disks with the index 1 for RAID md/0" is raised.