1

I manage a production server which has a RAID 1 array with two identical hard drives. Apologies in advance, I've read many threads on this but due to the critical nature of the server, it will save my job if someone can give me accurate step-by-step instructions. As stated, the server is 12.04.2 LTS. One of the drives failed and was automatically removed from the array. The hosting company replaced the faulty drive but did not rebuild the RAID array. Here is what I believe the relevant information from the server:

**df -h**
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        909G  775G   88G  90% /
udev            3.8G  4.0K  3.8G   1% /dev
tmpfs           1.6G  352K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.8G     0  3.8G   0% /run/shm
/dev/md0        458M   25M  409M   6% /boot

**cat /etc/fstab**
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md2 during installation
UUID=2d22fd63-9d2e-494c-92d9-89f411fb1b5d /               ext4    errors=remount-ro,usrquota 0       1
# /boot was on /dev/md0 during installation
UUID=b99dbfa5-fad2-4ba0-b741-70bd8ddff90e /boot           ext2    defaults        0       2
# swap was on /dev/md1 during installation
UUID=2d72a698-9c6c-4d81-9ed9-3d7ebe544e45 none            swap    sw              0       0

**cat /proc/mdstat** 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda2[0]
      499712 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sda3[0]
      7995840 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda4[0]
      968130304 blocks super 1.2 [2/1] [U_]

**mdadm -D /dev/md0**
/dev/md0:
        Version : 1.2
  Creation Time : Wed Jun 19 12:04:15 2013
     Raid Level : raid1
     Array Size : 499712 (488.08 MiB 511.71 MB)
  Used Dev Size : 499712 (488.08 MiB 511.71 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Apr 30 06:29:27 2015
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : paris086:0  (local to host paris086)
           UUID : ceba0bd5:ffa466c8:7e950165:71dc000d
         Events : 118

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       0        0        1      removed

**mdadm -D /dev/md1**
/dev/md1:
        Version : 1.2
  Creation Time : Wed Jun 19 12:04:15 2013
     Raid Level : raid1
     Array Size : 7995840 (7.63 GiB 8.19 GB)
  Used Dev Size : 7995840 (7.63 GiB 8.19 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Apr 30 13:04:46 2015
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : paris086:1  (local to host paris086)
           UUID : dd350924:63d6b46f:4cb885f3:11f2b430
         Events : 791

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       0        0        1      removed

**mdadm -D /dev/md2**
/dev/md2:
        Version : 1.2
  Creation Time : Wed Jun 19 12:04:15 2013
     Raid Level : raid1
     Array Size : 968130304 (923.28 GiB 991.37 GB)
  Used Dev Size : 968130304 (923.28 GiB 991.37 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Apr 30 13:35:25 2015
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : paris086:2  (local to host paris086)
           UUID : 2b8bd77e:7fc2806b:56ae349f:01473330
         Events : 539748

    Number   Major   Minor   RaidDevice State
       0       8        4        0      active sync   /dev/sda4
       1       0        0        1      removed

The first fail alert mdadm sent to me via email read:

A Fail event had been detected on md device /dev/md/2.

It could be related to component device /dev/sdb4.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1] sda2[0]
      499712 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb3[1](F) sda3[0]
      7995840 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sdb4[1](F) sda4[0]
      968130304 blocks super 1.2 [2/1] [U_]

The second fail event email read:

A Fail event had been detected on md device /dev/md/1.

It could be related to component device /dev/sdb3.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1] sda2[0]
      499712 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb3[1](F) sda3[0]
      7995840 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sdb4[1](F) sda4[0]
      968130304 blocks super 1.2 [2/1] [U_]

The third fail event email read:

A Fail event had been detected on md device /dev/md/0.

It could be related to component device /dev/sdb2.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1](F) sda2[0]
      499712 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sdb3[1](F) sda3[0]
      7995840 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sdb4[1](F) sda4[0]
      968130304 blocks super 1.2 [2/1] [U_]

unused devices: <none>

And from there on it has just been sending me daily "A DegradedArray event had been detected on md device" alerts for all three with the status [U_] on each.

Thank you for going through all this, any help would be much, much appreciated.

Some more information, from fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  1953525167   976762583+  ee  GPT

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/md2: 991.4 GB, 991365431296 bytes
2 heads, 4 sectors/track, 242032576 cylinders, total 1936260608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table

Disk /dev/md1: 8187 MB, 8187740160 bytes
2 heads, 4 sectors/track, 1998960 cylinders, total 15991680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md0: 511 MB, 511705088 bytes
2 heads, 4 sectors/track, 124928 cylinders, total 999424 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

And from parted -l

Model: ATA ST31000528AS (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  1018kB  1000kB                     bios_grub
 2      1018kB  513MB   512MB                      raid
 3      513MB   8705MB  8192MB                     raid
 4      8705MB  1000GB  991GB                      raid


Error: /dev/sdb: unrecognised disk label                                  

Model: Linux Software RAID Array (md)
Disk /dev/md2: 991GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  991GB  991GB  ext4


Model: Linux Software RAID Array (md)
Disk /dev/md1: 8188MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  8188MB  8188MB  linux-swap(v1)


Model: Linux Software RAID Array (md)
Disk /dev/md0: 512MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  512MB  512MB  ext2

1 Answer 1

1

I would use

sudo sgdisk /dev/sda -R /dev/sdb

to clone the partition table from /dev/sda to /dev/sdb.

Then,

sudo sgdisk /dev/sdb -G

to re-randomize the UUIDs on the new disk and avoid duplicate UUIDs.

Finally, add the new partitions to your RAID devices (I think the --remove step is necessary since the partitions are listed as failed parts of the current devices):

sudo mdadm --remove /dev/md0 /dev/sdb2
sudo mdadm --add /dev/md0 /dev/sdb2

Repeat for the other devices:

sudo mdadm --remove /dev/md1 /dev/sdb3
sudo mdadm --add /dev/md1 /dev/sdb3
sudo mdadm --remove /dev/md2 /dev/sdb4
sudo mdadm --add /dev/md2 /dev/sdb4
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .