Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have 6 HDDs in my box, 2x256Gb ATA OCZ-AGILITY4 SSDs.

  • Each is partitioned into one block of 48Gb and one block of 208Gb.
  • The 48Gb partitions are RAID0 striped and serve as the swap space (md0).
  • The 208Gb partitions are RAID1 striped and serve as the / file system (md1).

There are also 4x3.0Tb ATA ST3000DM001-9YN166 drives each with one partition, those 4 partitions are RAID5 striped (md128). All are ext4 formatted running ubuntu 12.04 server. sda and sdb are the SSDs, while sdc, sdd, sde, and sdf are the HDDs.

Randomly (as far as I can tell) md128 became read only. The sys log looks like this for the event:

Aug 23 16:25:24 crick kernel: [617040.416257] ata4.00: exception Emask 0x0 SAct 0x1f SErr 0x0 action 0x0
Aug 23 16:25:24 crick kernel: [617040.416260] ata4.00: irq_stat 0x40000008
Aug 23 16:25:24 crick kernel: [617040.416262] ata4.00: failed command: READ FPDMA QUEUED
Aug 23 16:25:24 crick kernel: [617040.416265] ata4.00: cmd 60/08:08:00:af:cc/00:00:9c:00:00/40 tag 1 ncq 4096 in
Aug 23 16:25:24 crick kernel: [617040.416265]          res 41/40:08:00:af:cc/00:00:9c:00:00/00 Emask 0x409 (media error) <F>
Aug 23 16:25:24 crick kernel: [617040.416266] ata4.00: status: { DRDY ERR }
Aug 23 16:25:24 crick kernel: [617040.416267] ata4.00: error: { UNC }
Aug 23 16:25:24 crick kernel: [617040.417510] ata4.00: configured for UDMA/133
Aug 23 16:25:24 crick kernel: [617040.417527] sd 3:0:0:0: [sdd] Unhandled sense code
Aug 23 16:25:24 crick kernel: [617040.417528] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417529] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 23 16:25:24 crick kernel: [617040.417530] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417531] Sense Key : Medium Error [current] [descriptor]
Aug 23 16:25:24 crick kernel: [617040.417533] Descriptor sense data with sense descriptors (in hex):
Aug 23 16:25:24 crick kernel: [617040.417534]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
Aug 23 16:25:24 crick kernel: [617040.417538]         9c cc af 00 
Aug 23 16:25:24 crick kernel: [617040.417540] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417541] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 23 16:25:24 crick kernel: [617040.417542] sd 3:0:0:0: [sdd] CDB: 
Aug 23 16:25:24 crick kernel: [617040.417543] Read(10): 28 00 9c cc af 00 00 00 08 00
Aug 23 16:25:24 crick kernel: [617040.417547] end_request: I/O error, dev sdd, sector 2630659840
Aug 23 16:25:24 crick kernel: [617040.417550] md/raid:md128: read error not correctable (sector 2630657792 on sdd1).
Aug 23 16:25:24 crick kernel: [617040.417552] md/raid:md128: Disk failure on sdd1, disabling device.
Aug 23 16:25:24 crick kernel: [617040.417552] md/raid:md128: Operation continuing on 2 devices.
Aug 23 16:25:24 crick kernel: [617040.417563] ata4: EH complete
Aug 23 16:25:25 crick kernel: [617040.455605] RAID conf printout:
Aug 23 16:25:25 crick kernel: [617040.455609]  --- level:5 rd:4 wd:2
Aug 23 16:25:25 crick kernel: [617040.455610]  disk 0, o:1, dev:sdc1
Aug 23 16:25:25 crick kernel: [617040.455611]  disk 1, o:0, dev:sdd1
Aug 23 16:25:25 crick kernel: [617040.455612]  disk 2, o:1, dev:sde1
Aug 23 16:25:25 crick kernel: [617040.489941] RAID conf printout:
Aug 23 16:25:25 crick kernel: [617040.489945]  --- level:5 rd:4 wd:2
Aug 23 16:25:25 crick kernel: [617040.489947]  disk 0, o:1, dev:sdc1
Aug 23 16:25:25 crick kernel: [617040.489948]  disk 2, o:1, dev:sde1

A slew of:
Aug 23 16:25:25 crick kernel: [617040.539926] Buffer I/O error on device md128, logical block 986401023
with different block addresses then

Aug 23 16:25:25 crick kernel: [617040.539929] EXT4-fs warning (device md128): ext4_end_bio:248: I/O error writing to inode 42993727 (offset 11551637504 size 524288 starting block 986400896)
Aug 23 16:25:25 crick kernel: [617040.541690] JBD2: Detected IO errors while flushing file data on md128-8
Aug 23 16:25:25 crick kernel: [617040.541707] Aborting journal on device md128-8.
Aug 23 16:25:25 crick kernel: [617040.541720] EXT4-fs error (device md128) in ext4_free_blocks:4702: Journal has aborted
Aug 23 16:25:25 crick kernel: [617040.541727] Buffer I/O error on device md128, logical block 1098416128
Aug 23 16:25:25 crick kernel: [617040.541729] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.541734] Buffer I/O error on device md128, logical block 0
Aug 23 16:25:25 crick kernel: [617040.541736] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.541740] JBD2: Error -5 detected when updating journal superblock for md128-8.
Aug 23 16:25:25 crick kernel: [617040.541743] EXT4-fs (md128): delayed block allocation failed for inode 49152114 at logical offset 994912 with max blocks 2048 with error -30
Aug 23 16:25:25 crick kernel: [617040.541745] EXT4-fs (md128): This should not happen!! Data will be lost
Aug 23 16:25:25 crick kernel: [617040.541745] 
Aug 23 16:25:25 crick kernel: [617040.541806] EXT4-fs (md128): previous I/O error to superblock detected
Aug 23 16:25:25 crick kernel: [617040.542518] EXT4-fs error (device md128) in ext4_da_writepages:2390: Journal has aborted
Aug 23 16:25:25 crick kernel: [617040.542526] JBD2: Detected IO errors while flushing file data on md128-8
Aug 23 16:25:25 crick kernel: [617040.542529] Buffer I/O error on device md128, logical block 0
Aug 23 16:25:25 crick kernel: [617040.542531] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.542798] EXT4-fs error (device md128): ext4_journal_start_sb:371: Detected aborted journal
Aug 23 16:25:25 crick kernel: [617040.542810] EXT4-fs (md128): Remounting filesystem read-only
Aug 23 16:25:25 crick kernel: [617040.542815] EXT4-fs (md128): previous I/O error to superblock detected
Aug 23 16:25:25 crick kernel: [617040.542835] EXT4-fs (md128): I/O error while writing superblock
Aug 23 16:25:25 crick kernel: [617040.542838] EXT4-fs (md128): ext4_da_writepages: jbd2_start: 15984 pages, ino 49152114; err -30
Aug 23 16:25:25 crick kernel: [617040.544887] EXT4-fs error (device md128): ext4_journal_start_sb:371: Detected aborted journal

When I use the disk utility or gparted sdf is gone, it's not even detected anymore. As such md128 is degraded. First thing I did was unmount it through the disk utility, then I tried seeing if anything else could find it:

fdisk -l

Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a88fe

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1       406368256   500117503    46874624   fd  Linux raid autodetect
/dev/sda2            2046   406368255   203183105    5  Extended
/dev/sda5            2048   406368255   203183104   fd  Linux raid autodetect

Partition table entries are not in disk order

Disk /dev/md0: 96.0 GB, 95998181376 bytes
2 heads, 4 sectors/track, 23437056 cylinders, total 187496448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 207.9 GB, 207925149696 bytes
2 heads, 4 sectors/track, 50762976 cylinders, total 406103808 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/sdb: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b0740

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *   406368256   500117503    46874624   fd  Linux raid autodetect
/dev/sdb2            2046   406368255   203183105    5  Extended
/dev/sdb5            2048   406368255   203183104   fd  Linux raid autodetect

Partition table entries are not in disk order

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/md128: 9001.4 GB, 9001370124288 bytes
2 heads, 4 sectors/track, -2097367168 cylinders, total 17580801024 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1572864 bytes
Disk identifier: 0x00000000

Disk /dev/md128 doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

cat /etc/fstab

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md127 during installation
UUID=b63b7341-0b85-40ed-a67b-acfe4f65f563 /               ext4    errors=remount-ro 0       1
# /genome was on /dev/md128 during installation
UUID=bd6b54be-12ca-479d-879e-8d788fa9d039 /genome         ext4    defaults        0       2
# swap was on /dev/md126 during installation
UUID=0470550a-6e92-485d-ad41-665c3f313287 none            swap    sw              0       0

~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md128 : active raid5 sde1[2] sdc1[0]
      8790400512 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [U_U_]

md1 : active raid1 sdb5[1] sda5[0]
      203051904 blocks super 1.2 [2/2] [UU]

md0 : active raid0 sdb1[1] sda1[0]
      93748224 blocks super 1.2 512k chunks

unused devices: <none>

~# mdadm --detail --scan

ARRAY /dev/md/0 metadata=1.2 name=crick:0 UUID=d0d08eab:a7e54021:25973acb:10dd5fba
ARRAY /dev/md/1 metadata=1.2 name=crick:1 UUID=e2774945:5b3ee3eb:2ad9390f:35153b82
ARRAY /dev/md/128 metadata=1.2 name=crick:128 UUID=345cb755:0ae1c919:d98a45ca:1baf3364

This is where my understanding of linux is not the greatest though. From what I can tell the array superblock sees the UUID for sdf that it was built on, but (for whatever reason) sdf isn't being detected. This means that the RAID array is broken and so ubuntu set the file system to RO as a safety precaution. I can still open, edit, and create files on / with no issue, the OS still runs perfectly, I just can't access the data on md128. I haven't turned off the machine yet to remove the drive to see if it's failed or not.

About a week ago the same issue (random switch to RO) occured, only that time it hit md1 (the SSDs). When that happened the superblock was erased on md128. I got lucky when reinstalling the OS and had it rebuild the RAID using the existing partitions (IE didn't format the partitions), and managed to recover 99.9% of the data (the files that were being accessed when it happened were corrupted).

  • Is the current issue with md128 a problem that I managed to band-aid over until now or is this a sign of some deeper issue that I should be worried about?
  • What is the best way to move on from here: replace sdf and restripe into md128, replace sdf and tear down/rebuild md128, or something else?

EDIT: Sorry for the OP, I'm new to the forum and didn't read the formatting instructions, I've corrected it now

share|improve this question
add comment

1 Answer

up vote 0 down vote accepted

Most likely sdf died some time ago and you did not notice. Now sdd has some bad sectors. The failed reads cause the filesystem to switch to read only. You need to replace the failed drive, and fsck the filesystem.

share|improve this answer
    
Sorry about the extremely delayed reply. This got me up and running again. I'd upvote, but I don't have a high enough reputation. Thanks for the help! –  Seth_m55 Nov 25 '13 at 20:56
    
@Seth_m55, click the check box to accept it as the correct answer. –  psusi Nov 25 '13 at 21:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.