Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I've been getting some RAID5 failures for some time when under heavy load. I've always been able to force reassemble the array after a reboot, and the Event count between the dropped disks and the non-dropped one is always less than 20.

After some investigation, I think I figured that the issue isn't on the disk themselves (they usually drop together at the same time) but on the controller.

Here's an extract of the dmesg log when this happens (in that case, it happened while forcing a check on the array): http://paste.ubuntu.com/6067736/

After it failed like this, both drives /dev/sde and /dev/sdf seem to be inaccessible, as a smartctl -a /dev/sde gives up this:

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-37-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               /5:0:0:0
Product:              
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Even though a 600PB disk would be nice, it's just 1.5TB instead.

I have 3 of the raid disks that are connected to ATA ports on the motherboard, and 2 disks are connected on an external (PCIex) ATA controller. From the failures in the log I have reason to believe it's the controller that has issues.

What could I do to fix this? Change ATA cables? Change some settings? I'm pretty new to linux.

Data from smartctl:

Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD15EARS-00MVWB0
Serial Number:    WD-WMAZA2091111
LU WWN Device Id: 5 0014ee 6ab49271f
Firmware Version: 51.0AB51

sdf gives out those stats:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   171   169   051    Pre-fail  Always       -       34595
  3 Spin_Up_Time            0x0027   253   253   021    Pre-fail  Always       -       1016
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       78
  5 Reallocated_Sector_Ct   0x0033   162   162   140    Pre-fail  Always       -       735
  7 Seek_Error_Rate         0x002e   200   199   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   070   070   000    Old_age   Always       -       22178
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       76
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       43
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3415
194 Temperature_Celsius     0x0022   118   110   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   001   001   000    Old_age   Always       -       691
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       6
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       2
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   199   189   000    Old_age   Offline      -       319

While sde doesn't show any reallocations or read errors:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   253   253   021    Pre-fail  Always       -       925
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       73
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   070   070   000    Old_age   Always       -       22178
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       71
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       38
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3378
194 Temperature_Celsius     0x0022   114   106   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

Disks are indeed green, didn't think it would be a problem when I bought them.

What seems strange to me is that under load usually both disks just drop out of the array. Could the faulty disk somehow "cascade" to the other one on the same ATA controller?

In any case I guess I have to buy at least one new drive.

Edit: funny that I said usually both drives die at the same time; after checking this morning I just realized only sdf dropped of the array, and it's been for quite some time as the even count difference is about 4000. In that case I assume it makes no sense to try to add it back to the array, I need to find a spare one quickly.

share|improve this question
    
dmesg pretty clearly says the drive is toast ( medium errors ). What is the SMART status after you reboot? –  psusi Sep 6 '13 at 0:41

1 Answer 1

The ATA Errors are fairly generic, it is easier to pick up the errors in Linux to debug.

There are two main things that cause the ATA Errors:

  1. Disks are faulty/green
  2. Communication issue from the disk to the Processor/Memory (SATA cables or connectors they are connecting to)

This is more common with the disk issues from my experience.

  • If you have green drives will be an issue for the nature of some of the drives' nature to save power. Some model of drives will spin down, and when the RAID tries to access the disk it will flag it as an ATA error if the drives does not respond in time. (This can be a false negative which there is not much you can do.)
  • If there is a bad disk, you may need to test the disk in another computer with the drive manufacture's tools. This would make sure it is not the connection and a deeper test of the drive.

If you had a cable to switch out, it may help rule that issues out.

Interestingly enough, I have seen this issue on WDC. (I think there was a firmware that may have addressed it on the drive... Which I don't think they make it publicly easy to do anymore.)

What is the drive model?

share|improve this answer
    
Here are the smartctl output on both drives after a reboot: –  user190443 Sep 6 '13 at 7:25
    
The Reallocated_Sector_Ct is an issue on the spindle disk. That disk will need replaced. The Offline_Uncorrectable is an issue where the drive is not responding and this should not happen even on the green drives. Green drives are typically not recommended from the drive manufacture to have the disks in RAID. This is primarily due to the spin downs on the disks and nature of RAID needing to check on the disks periodically. –  Mhynlo Sep 6 '13 at 20:45
1  
WD green drives don't spin down unless you ask them to. The raid time out thing only applies to the drive spending a long time retrying a bad sector and there is a knob somewhere in the kernel you can adjust to give it more time. –  psusi Sep 13 '13 at 14:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.