mdadm fails after 10.04 -> 12.04 upgrade

Question

I just upgraded my Ubuntu server (in the role of NAS) from 10.04 to 12.04. Since both are LTS versions I didn't expect any issues and... didn't make any backups. My bad.

After upgrade, the server didn't come up, grub rescue greeted me.

Originaly I had 4 RAID arrays: One, 10GB for system /dev/md0, was assembled from /dev/sd[bcdef]1 in RAID1 mode. Second, 10GB for cache, /dev/md1, was assembled from /dev/sd[bcdef]2 in RAID1(?) mode. Third, /dev/md5, was assembled from /dev/sd[bcdef]6 in RAID5 mode. All partitions 500GB each. Last, /dev/md6, was assembled from /dev/sd[bcdef]5 in RAID6 mode. All partitions 500GB each.

Since I don't have 5TB (or even 3.5TB) of mediums to make a backup, I installed a clean Ubuntu server system on /dev/sda using minimal cd. I wasn't able to boot eighter, but it could be fault on part of my hardware, which is quite dated and doesn't have the option to boot from different hard drive. So I run XUbuntu live cd, and from it I was able to mount one of the system partitions, and made a full backup of system files. Then I installed the 12.04 on properly recognized during setup /dev/sd[bcdef]1 RAID1. Unfortunately, the system actually installed solely on /dev/sde1, ignoring the raid layout. After that, the system didn't boot eigher. Afraid to touch anything more, I grabbed the XUbuntu live cd again, and from it I could muster the following diagnostic data.

I am interested in recovering only /dev/md5 and /dev/md6.

It looks, like the installer messed with the superblocks on /dev/sde. I hope it is not fatal...

#sudo /usr/share/mdadm/mkconf
...

# definitions of existing MD arrays
ARRAY /dev/md124 UUID=01b9a5fe:1be857d7:c20db638:73a0ec38
ARRAY /dev/md127 UUID=00000000:00000000:00000000:00000000
   spares=2
ARRAY /dev/md/6 metadata=1.2 UUID=2a8fd2df:3ad53552:e8b80ecf:a46d93a2 name=pieciodysk:6
ARRAY /dev/md/5 metadata=1.2 UUID=a3945c40:73237cd6:3d61998e:8f773d03 name=pieciodysk:5
ARRAY /dev/md127 UUID=00000000:00000000:00000000:00000000
   spares=4
ARRAY /dev/md/6 metadata=1.2 UUID=2a8fd2df:3ad53552:e8b80ecf:a46d93a2 name=pieciodysk:6
   spares=4
ARRAY /dev/md/5 metadata=1.2 UUID=a3945c40:73237cd6:3d61998e:8f773d03 name=pieciodysk:5
   spares=4

fdsik /dev/sdb:

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c3e8a

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048    19531775     9764864   fd  Linux raid autodetect
/dev/sdb2        19531776    21485567      976896   fd  Linux raid autodetect
/dev/sdb3        21485568  1953510662   966012547+   5  Extended
/dev/sdb5        21485631   987490082   483002226   fd  Linux raid autodetect
/dev/sdb6       987490146  1953510662   483010258+  fd  Linux raid autodetect

fdsik /dev/sdc:

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          63    19535039     9767488+  fd  Linux raid autodetect
/dev/sdc2        19535040    21494969      979965   fd  Linux raid autodetect
/dev/sdc3        21494970  1953520064   966012547+   5  Extended
/dev/sdc5        21495033   987499484   483002226   fd  Linux raid autodetect
/dev/sdc6       987499548  1953520064   483010258+  fd  Linux raid autodetect

fdsik /dev/sdd:

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1   *        2048    19531775     9764864   fd  Linux raid autodetect
/dev/sdd2        19531776    21485567      976896   fd  Linux raid autodetect
/dev/sdd3        21485568  1953510662   966012547+   5  Extended
/dev/sdd5        21485631   987490082   483002226   fd  Linux raid autodetect
/dev/sdd6       987490146  1953510662   483010258+  fd  Linux raid autodetect

fdsik /dev/sde:

Disk /dev/sde: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1   *          63    19535039     9767488+  fd  Linux raid autodetect
/dev/sde2        19535040    21494969      979965   fd  Linux raid autodetect
/dev/sde3        21494970  2930272064  1454388547+   5  Extended
/dev/sde5        21495033   987499484   483002226   fd  Linux raid autodetect
/dev/sde6       987499548  1953520064   483010258+  fd  Linux raid autodetect
/dev/sde7      1953520128  2930272064   488375968+  83  Linux

fdsik /dev/sdf:

Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x3743db4a

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1              63    19535039     9767488+  fd  Linux raid autodetect
/dev/sdf2        19535040    21494969      979965   fd  Linux raid autodetect
/dev/sdf3        21494970  1953520064   966012547+   5  Extended
/dev/sdf5        21495033   987499484   483002226   fd  Linux raid autodetect
/dev/sdf6       987499548  1953520064   483010258+  fd  Linux raid autodetect

/dev/sda is an auxilary disk, which is not used for anything in particular.

sudo mdadm --examine /dev/sdb5

 /dev/sdb5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2a8fd2df:3ad53552:e8b80ecf:a46d93a2
           Name : pieciodysk:6
  Creation Time : Sun Sep 12 19:36:21 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966004180 (460.63 GiB 494.59 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 64b1c028:14da81bd:06bfb40b:c30ff7d6

    Update Time : Tue Sep 11 17:05:27 2012
       Checksum : e4b0a0dd - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdc5

 /dev/sdc5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2a8fd2df:3ad53552:e8b80ecf:a46d93a2
           Name : pieciodysk:6
  Creation Time : Sun Sep 12 19:36:21 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966004180 (460.63 GiB 494.59 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 6b8374d4:4a4f71a1:23f7fa5c:7a7953c1

    Update Time : Tue Sep 11 17:05:27 2012
       Checksum : aff689f7 - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdd5

 /dev/sdd5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2a8fd2df:3ad53552:e8b80ecf:a46d93a2
           Name : pieciodysk:6
  Creation Time : Sun Sep 12 19:36:21 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966004180 (460.63 GiB 494.59 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 017925d6:8db5b2ca:8795d93e:c0205358

    Update Time : Tue Sep 11 17:05:27 2012
       Checksum : 53c72b78 - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sde5

 /dev/sde5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 2a8fd2df:3ad53552:e8b80ecf:a46d93a2
           Name : pieciodysk:6
  Creation Time : Sun Sep 12 19:36:21 2010
     Raid Level : raid6
   Raid Devices : 5

 Avail Dev Size : 966004180 (460.63 GiB 494.59 GB)
     Array Size : 2898012288 (1381.88 GiB 1483.78 GB)
  Used Dev Size : 966004096 (460.63 GiB 494.59 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a9a93188:1c1ae289:b56398c7:6be226cb

 Internal Bitmap : 2 sectors from superblock
    Update Time : Mon Sep 10 08:31:33 2012
       Checksum : c0a8031a - correct
         Events : 80390

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAAA ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdf5

 /dev/sdf5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2a8fd2df:3ad53552:e8b80ecf:a46d93a2
           Name : pieciodysk:6
  Creation Time : Sun Sep 12 19:36:21 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966004180 (460.63 GiB 494.59 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : d7b87b43:298cdd3d:3f900370:198f3f48

    Update Time : Tue Sep 11 17:05:27 2012
       Checksum : 555eaafb - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdb6

 /dev/sdb6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a3945c40:73237cd6:3d61998e:8f773d03
           Name : pieciodysk:5
  Creation Time : Sun Sep 12 19:36:39 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966020245 (460.63 GiB 494.60 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : a61e9d6f:35b53cb1:96105de8:60991128

    Update Time : Tue Sep 11 17:05:26 2012
       Checksum : 8aa72e70 - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdc6

 /dev/sdc6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a3945c40:73237cd6:3d61998e:8f773d03
           Name : pieciodysk:5
  Creation Time : Sun Sep 12 19:36:39 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966020245 (460.63 GiB 494.60 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 2006abf7:13538568:e5cf2d58:2961a894

    Update Time : Tue Sep 11 17:05:26 2012
       Checksum : a6653ae8 - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdd6

 /dev/sdd6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a3945c40:73237cd6:3d61998e:8f773d03
           Name : pieciodysk:5
  Creation Time : Sun Sep 12 19:36:39 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966020245 (460.63 GiB 494.60 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 1cf3988f:5259d109:9a8cd419:db31b135

    Update Time : Tue Sep 11 17:05:26 2012
       Checksum : 424ebb86 - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sde6

 /dev/sde6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : a3945c40:73237cd6:3d61998e:8f773d03
           Name : pieciodysk:5
  Creation Time : Sun Sep 12 19:36:39 2010
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 966020245 (460.63 GiB 494.60 GB)
     Array Size : 3864080896 (1842.54 GiB 1978.41 GB)
  Used Dev Size : 966020224 (460.63 GiB 494.60 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a496841f:489b48dc:e330989a:5991ff4f

 Internal Bitmap : 2 sectors from superblock
    Update Time : Mon Sep 10 08:31:33 2012
       Checksum : 3fd62187 - correct
         Events : 66542

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAAA ('A' == active, '.' == missing)

sudo mdadm --examine /dev/sdf6

 /dev/sdf6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a3945c40:73237cd6:3d61998e:8f773d03
           Name : pieciodysk:5
  Creation Time : Sun Sep 12 19:36:39 2010
     Raid Level : -unknown-
   Raid Devices : 0

 Avail Dev Size : 966020245 (460.63 GiB 494.60 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 36e0774e:6d6fbde9:d979fbe2:9d91b3ba

    Update Time : Tue Sep 11 17:05:26 2012
       Checksum : 2f430bc0 - correct
         Events : 1


   Device Role : spare
   Array State :  ('A' == active, '.' == missing)

Answer 1

The problem came from combination of two events:

1. The known bug in kernel used by Ubuntu installer mini-cd, which effectively wiped down the superblock on raid members (actually excluding one, /dev/sde. I guess it is because of one problem of Ubuntu 12.04 with racing condition during boot that prevents some disks to be visible to mdadm - see this link).
2. Another bug in kernel used by Ubuntu 12.04 kernel (which may affect others), which effectively reset superblocks on raid members after upgrade.

The solution:

1. Recover the raid members proper order from syslog taken from system when it was healthy (or be prepared to permute the order in the following step as long as the raid is restored)
2. Issue mdadm -C /dev/md5 -e 1.2 -l 5 -n 5 /dev/sd{b,f,c,e,d}6 --assume-clean --uuid=a3945c40:73237cd6:3d61998e:8f773d03 and mdadm -C /dev/md6 -e 1.2 -l 6 -n 5 /dev/sd{b,f,c,e,d}5 --assume-clean --uuid=2a8fd2df:3ad53552:e8b80ecf:a46d93a2 as root from Ubuntu 10.04 (not newer) live CD. (Of course names of member devices, UUIDs, number of raid devices etc. will be different in your setup.)
3. If contents of raids is valid - then we are done. No need to use the obsolete 10.04 any longer.

For further info see this Server Fault question: What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?