0

I need help with my raid5 after it crashed through bad sectors. Previously I could re-assemble it with

# mdadm --assemble --force -v /dev/md0 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1

but while doing a backup it crashed again and now I'm not able to re-assemble it anymore because two disks are out-of-date:

# mdadm --assemble --force -v /dev/md0 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdg1 to /dev/md0 as 1
mdadm: added /dev/sdh1 to /dev/md0 as 2 (possibly out of date)
mdadm: added /dev/sdi1 to /dev/md0 as 3 (possibly out of date)
mdadm: failed to add /dev/sde1 to /dev/md0: Device or resource busy
mdadm: added /dev/sdf1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.

As you can see here only two devices sdf1 and sdg1 have set the latest update time (+ spare, but it's not finished rebuilding).

 mdadm --examine /dev/sd[efghi]1 | egrep 'dev|Update|Role|State|Chunk Size'
/dev/sde1:
          State : clean
    Update Time : Sun May 10 04:15:59 2015
     Chunk Size : 512K
   Device Role : spare
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdf1:
          State : clean
    Update Time : Sun May 10 04:15:59 2015
     Chunk Size : 512K
   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdg1:
          State : clean
    Update Time : Sun May 10 04:15:59 2015
     Chunk Size : 512K
   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdh1:
          State : clean
    Update Time : Sat May  9 23:10:06 2015
     Chunk Size : 512K
   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdi1:
          State : active
    Update Time : Sat Dec  7 12:43:00 2013
     Chunk Size : 512K
   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

I did not change any data on the raid since the update time from the other two devices sdh1 and sdi1. I don't need to resync all data, I just need to backup the last files, so I just need to mount it a last time read-only.

Is there any way to do this? Maybe I can force it to ignore the out-of-date? I'm wondering why --force doesn't work anymore...

Complete information to the raid5 devices:

# mdadm --examine /dev/sd[efghi]1
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a87dfb70:2ecd03f9:ee62b434:fc637218
           Name : m08002-lin:data2gb
  Creation Time : Mon Sep  2 12:48:02 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : 624d3873:7970ba27:da0f511a:45367bdd

    Update Time : Sun May 10 04:15:59 2015
       Checksum : 599a5235 - correct
         Events : 108804

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a87dfb70:2ecd03f9:ee62b434:fc637218
           Name : m08002-lin:data2gb
  Creation Time : Mon Sep  2 12:48:02 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : 4827a499:12980366:0de13b87:541a9b5e

    Update Time : Sun May 10 04:15:59 2015
       Checksum : ac5a08f2 - correct
         Events : 108804

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a87dfb70:2ecd03f9:ee62b434:fc637218
           Name : m08002-lin:data2gb
  Creation Time : Mon Sep  2 12:48:02 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : 8c959d62:3b9c1eac:6f8d7d92:13454ab4

    Update Time : Sun May 10 04:15:59 2015
       Checksum : 1c5f5282 - correct
         Events : 108804

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a87dfb70:2ecd03f9:ee62b434:fc637218
           Name : m08002-lin:data2gb
  Creation Time : Mon Sep  2 12:48:02 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : df6b9eab:ea3c6e3a:47858e6d:1eb0783d

    Update Time : Sat May  9 23:10:06 2015
       Checksum : 57f1e4b2 - correct
         Events : 108796

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a87dfb70:2ecd03f9:ee62b434:fc637218
           Name : m08002-lin:data2gb
  Creation Time : Mon Sep  2 12:48:02 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
     Array Size : 5860145664 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : active
    Device UUID : fbc64ec7:a97a36c1:69cc3812:37878af1

    Update Time : Sat Dec  7 12:43:00 2013
       Checksum : 507acca4 - correct
         Events : 83904

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
Improve this question

2 Answers 2

Reset to default
0

I could access my raid and backup my files!!

First I removed the spare device. Then I checked the other devices. There were two devices with bad sectors and I knew that the raid will crash again while resyncing when it wants to read or write to bad sectors. So I decided to clear the raid and build up a new degraded raid that is not syncing and still has access to all previous data. I cleared all superblocks and created the raid leaving out one damaged device with

mdadm --stop /dev/md127 (or /dev/md0)
mdadm --zero-superblock /dev/sd[efgh]1
mdadm --create /dev/md127 --level=5 --raid-devices=4 --assume-clean /dev/sde1 /dev/sdf1 /dev/sdg1 missing

Important is the original order of the devices for the original raid and to create the new raid in the same order as well as to use the parameter --assume-clean! You can get the original order with

mdadm --examine /dev/sd[efghi]1

Have a look at Device Role.

After re-creating the raid using assume-clean I could mount md127 and access all data directly without doing anything else.

Improve this answer
1
  • This is, however, risky: I tried this and the --create command assumed a wrong size for the partition and thus the result was utter gibberish. A way to make --assemble and --force work is how I saved it.
    – Zds
    Apr 10 at 2:41
0

I had this same problem and the --zero-superblock and --create resulted in complete gibberish since it auto-guessed the partition sizes wrong and thus luks partition on top of it was unreadable.

Following answer here saved me: https://serverfault.com/a/1147308/85383

The trick was to set the md_mod.start_dirty_degraded kernel parameter to 1. This obviously can still lose you some blocks that were on in sync when the array went down, but apart from that, this worked for me. It was an ext3 partition anyway, so somewhat robust.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .