I manage a production server which has a RAID 1 array with two identical hard drives. Apologies in advance, I've read many threads on this but due to the critical nature of the server, it will save my job if someone can give me accurate step-by-step instructions. As stated, the server is 12.04.2 LTS. One of the drives failed and was automatically removed from the array. The hosting company replaced the faulty drive but did not rebuild the RAID array. Here is what I believe the relevant information from the server:
**df -h**
Filesystem Size Used Avail Use% Mounted on
/dev/md2 909G 775G 88G 90% /
udev 3.8G 4.0K 3.8G 1% /dev
tmpfs 1.6G 352K 1.6G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.8G 0 3.8G 0% /run/shm
/dev/md0 458M 25M 409M 6% /boot
**cat /etc/fstab**
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/md2 during installation
UUID=2d22fd63-9d2e-494c-92d9-89f411fb1b5d / ext4 errors=remount-ro,usrquota 0 1
# /boot was on /dev/md0 during installation
UUID=b99dbfa5-fad2-4ba0-b741-70bd8ddff90e /boot ext2 defaults 0 2
# swap was on /dev/md1 during installation
UUID=2d72a698-9c6c-4d81-9ed9-3d7ebe544e45 none swap sw 0 0
**cat /proc/mdstat**
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda2[0]
499712 blocks super 1.2 [2/1] [U_]
md1 : active raid1 sda3[0]
7995840 blocks super 1.2 [2/1] [U_]
md2 : active raid1 sda4[0]
968130304 blocks super 1.2 [2/1] [U_]
**mdadm -D /dev/md0**
/dev/md0:
Version : 1.2
Creation Time : Wed Jun 19 12:04:15 2013
Raid Level : raid1
Array Size : 499712 (488.08 MiB 511.71 MB)
Used Dev Size : 499712 (488.08 MiB 511.71 MB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Thu Apr 30 06:29:27 2015
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : paris086:0 (local to host paris086)
UUID : ceba0bd5:ffa466c8:7e950165:71dc000d
Events : 118
Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 0 0 1 removed
**mdadm -D /dev/md1**
/dev/md1:
Version : 1.2
Creation Time : Wed Jun 19 12:04:15 2013
Raid Level : raid1
Array Size : 7995840 (7.63 GiB 8.19 GB)
Used Dev Size : 7995840 (7.63 GiB 8.19 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Thu Apr 30 13:04:46 2015
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : paris086:1 (local to host paris086)
UUID : dd350924:63d6b46f:4cb885f3:11f2b430
Events : 791
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 0 0 1 removed
**mdadm -D /dev/md2**
/dev/md2:
Version : 1.2
Creation Time : Wed Jun 19 12:04:15 2013
Raid Level : raid1
Array Size : 968130304 (923.28 GiB 991.37 GB)
Used Dev Size : 968130304 (923.28 GiB 991.37 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Thu Apr 30 13:35:25 2015
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : paris086:2 (local to host paris086)
UUID : 2b8bd77e:7fc2806b:56ae349f:01473330
Events : 539748
Number Major Minor RaidDevice State
0 8 4 0 active sync /dev/sda4
1 0 0 1 removed
The first fail alert mdadm sent to me via email read:
A Fail event had been detected on md device /dev/md/2.
It could be related to component device /dev/sdb4.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1] sda2[0]
499712 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdb3[1](F) sda3[0]
7995840 blocks super 1.2 [2/1] [U_]
md2 : active raid1 sdb4[1](F) sda4[0]
968130304 blocks super 1.2 [2/1] [U_]
The second fail event email read:
A Fail event had been detected on md device /dev/md/1.
It could be related to component device /dev/sdb3.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1] sda2[0]
499712 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdb3[1](F) sda3[0]
7995840 blocks super 1.2 [2/1] [U_]
md2 : active raid1 sdb4[1](F) sda4[0]
968130304 blocks super 1.2 [2/1] [U_]
The third fail event email read:
A Fail event had been detected on md device /dev/md/0.
It could be related to component device /dev/sdb2.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1](F) sda2[0]
499712 blocks super 1.2 [2/1] [U_]
md1 : active raid1 sdb3[1](F) sda3[0]
7995840 blocks super 1.2 [2/1] [U_]
md2 : active raid1 sdb4[1](F) sda4[0]
968130304 blocks super 1.2 [2/1] [U_]
unused devices: <none>
And from there on it has just been sending me daily "A DegradedArray event had been detected on md device" alerts for all three with the status [U_] on each.
Thank you for going through all this, any help would be much, much appreciated.
Some more information, from fdisk -l
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 1 1953525167 976762583+ ee GPT
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/md2: 991.4 GB, 991365431296 bytes
2 heads, 4 sectors/track, 242032576 cylinders, total 1936260608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md2 doesn't contain a valid partition table
Disk /dev/md1: 8187 MB, 8187740160 bytes
2 heads, 4 sectors/track, 1998960 cylinders, total 15991680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/md0: 511 MB, 511705088 bytes
2 heads, 4 sectors/track, 124928 cylinders, total 999424 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
And from parted -l
Model: ATA ST31000528AS (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 17.4kB 1018kB 1000kB bios_grub
2 1018kB 513MB 512MB raid
3 513MB 8705MB 8192MB raid
4 8705MB 1000GB 991GB raid
Error: /dev/sdb: unrecognised disk label
Model: Linux Software RAID Array (md)
Disk /dev/md2: 991GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 991GB 991GB ext4
Model: Linux Software RAID Array (md)
Disk /dev/md1: 8188MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 8188MB 8188MB linux-swap(v1)
Model: Linux Software RAID Array (md)
Disk /dev/md0: 512MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 512MB 512MB ext2