I'm going to need some expert help - free beer/coffee to anyone who get's me on my way!

My System I'm running Ubuntu 11.10 with a 3 disk software RAID 5 configuration, with 3TB hard drives, in ext4 format.

What I Did I recently reviewed the health of the disks in the Disk Utility, and saw that two of my drives had bad sectors. Everything was working fine, but I got a little worried. When I looked at the mdadm --detail, I could see that one of the drives was in failure, and the raid was running degraded. However, one of the drives with a few failed sectors was in fact not reporting a failure by mdadm.

I bought a new hard drive, and followed the steps to replace a failed drive in a RAID 5 software configuration. (I marked the drive as failed, removed it, turned off computer, replaced it, partitioned the new drive, and added it back to the RAID). The process nearly completed (I think), but when I reviewed mdstat detail again, things looked far worse than they were before.

What I Have Now

Here is the output of mdstat --detail /dev/mda127

root@mediapc:/home/jason# mdadm --detail /dev/md127 
        Version : 1.2
  Creation Time : Sun Mar 10 08:57:16 2013
     Raid Level : raid5
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 2930265088 (2794.52 GiB 3000.59 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

Update Time : Sun Mar 16 06:48:06 2014
      State : clean, FAILED
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1

     Layout : left-symmetric
 Chunk Size : 512K

       Name : mediapc:127  (local to host mediapc)
       UUID : ffbed825:f397afb1:86535cd8:64f8c314
     Events : 373600

Number   Major   Minor   RaidDevice State
   0       8        1        0      active sync   /dev/sda1
   1       0        0        1      removed
   2       0        0        2      removed

   3       8       32        -      faulty spare   /dev/sdc
   4       8       17        -      spare   /dev/sdb1

My newly added drive (dev/sdb1) was added as a spare. The partition that was reporting some sector issues (in Disk Utility) but was working ok in RAID is now reporting as a faulty spare.

I'm pretty sure all/most of the data is still present, I just have no way to get to it. I'll buy more drives (external) if anyone has an idea on how to get my data back, or how to get my raid running again, if at all possible.

I wanted to try and fix the sectors on the now-faulty spare drive (/dev/sdc) but fsck fails a couple of different ways:

root@mediapc:/home/jason# fsck /dev/sdc
fsck from util-linux 2.19.1
fsck: fsck.linux_raid_member: not found
fsck: Error 2 while executing fsck.linux_raid_member for /dev/sdc


root@mediapc:/home/jason# fsck /dev/sdc1
fsck from util-linux 2.19.1
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext4: Device or resource busy while trying to open /dev/sdc1
Filesystem mounted or opened exclusively by another program?

I tried added both drives back to the array as they are, but both fail to add:

root@mediapc:/home/jason# mdadm --add /dev/md127 /dev/sdb1
mdadm: Cannot open /dev/sdb1: Device or resource busy

I tried search on way to get fsck to run, but I didn't want to reboot the system until I was sure that was the next correct move.

Really looking for some expert help. I want my data, and I want to get this RAID 5 working - but first and foremost I'd like the data. Thank you in advance. Let me know if there is anything else I can provide.

Regards, Jason


You'll need to umount the RAID-array before you can run fsck on any of the devices that are part of it.

I really hope you have a backup, cause it looks like you're heading for a crash.
Trying to recover from a single disk failure increases the chance of other disks in the array to fail as well due to the fact that the load on them increases during the repair. (ref: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162)

  • Thank you. I had previously unmounted the raid drive. In Disk Utility, I see the option to "check Filesystem" on both the failed spare, as well as the RAID partion. On both, I get the error message: "File System is NOT clean". Anything else I can do or try?? – user258779 Mar 16 '14 at 19:42
  • Sorry, I'm not familiar with Disk Utility. Is there no option to "repair" the filesystem? You might want to try running badblocks on the device(s). Mind you, that will take a long time. – Mausy5043 Mar 20 '14 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.