Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a mdadm RAID-6 in my home server of 5x1Tb WD Green HDDs. Read speed is more than enough - 268 Mb/s in dd. But write speed is just 37.1 Mb/s. (Both tested via dd on 48Gb file, RAM size is 1Gb, block size used in testing is 8kb)

Could you please suggest why write speed is so low and is there any ways to improve it? CPU usage during writing is just 25% (i.e. half of 1 core of Opteron 165) No business critical data there & server is UPS-backed.

mdstat is:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sda1[0] sdd1[4] sde1[3] sdf1[2] sdb1[1]
      2929683456 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [5/5] [UUUUU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>

Any suggestions?

Things like writeback, barrier, nobh didn't helped. DD blocksize=1M, 8M didn't changed anything. It looks like mdadm physically reads sectors to calculate parity even if it does not matter... Is that correct?

Update: Speed degradation after altered stripe cache was actually because 1 HDD probably failed during testing, nice :-D

Resolved: After increasing stripe cache & switching to external bitmap, my speeds are 160 Mb/s writes, 260 Mb/s reads. :-D

share|improve this question
    
Hey Bars.. What have you tweaked and after what change did the performance drop down to 120Mb/s? I am having a similar issue but my performance is way worse :( Any suggestions would be useful! Thanks! –  user12968 Mar 24 '11 at 23:51
    
In my case I had a diskfail in the middle of optimization :-) –  BarsMonster Mar 25 '11 at 4:16
add comment

2 Answers

up vote 7 down vote accepted

Have you tried tuning /sys/block/mdX/md/stripe_cache_size?

According to this forum post (in Norwegian, sorry) "tuning this parameter is more essential the more disks and the faster system you have":

On my system I get the best performance using the value 8192. If I use the default value of 256 the write performance drops 66%.

Quoting his speed for comparison:

Disks: 8xSeagate 2TB LP (5900RPM) in mdadm RAID6 (-n 512) (stripe_size_cache=8192).

CPU: Intel X3430 (4x2.4GHz, 8GB DDR3 ECC RAM)

Speed: 387 MB/s sequential write, 704 MB/s sequential read, 669 random seeks per sec.

My home server has almost the same disks as you, using RAID 5:

Disks: 4x1.5TB WD Green in RAID 5 (stripe_size_cache=256 - the default)

CPU: Intel i7 920 (2.66 GHz, 6 GB RAM)

Speed: 60 MB/s sequential write, 138 MB/s sequential read (according to Bonnie++)

So it looks like sequential write performance is around 50% of read performance.

For what performance to expect, the Linux Raid Wiki says about RAID 5:

Reads are almost similar to RAID-0 reads, writes can be either rather expensive (requiring read-in prior to write, in order to be able to calculate the correct parity information, such as in database operations), or similar to RAID-1 writes (when larger sequential writes are performed, and parity can be calculated directly from the other blocks to be written).

And about RAID 6:

Read performance is similar to RAID-5 but write performance is worse.

share|improve this answer
    
Write speed dropped to 15-20 Mb/s, read speed down to 110 :-S This just does not make sence :-( –  BarsMonster Dec 29 '10 at 23:06
    
Try some other values, see what happens. Not an expert on this, but I have WD Greens (4x1.5TB) in md RAID 5 with the same issue - rather slow write speed. My system is down at the moment, but I'll experiment at my end as well, and update here if I manage to improve it. –  j-g-faustus Dec 29 '10 at 23:13
    
BTW: Do you have AHCI enabled? Which file system - ext3, ext4? –  j-g-faustus Dec 29 '10 at 23:20
    
Not sure about AHCI, it's very hard to take a look at BIOS. ext4. –  BarsMonster Dec 29 '10 at 23:30
    
Damn, stuck with 120Mb/s reading speed, can't get back to my 268 :-( This is so frustrating... I don't see why can't mdadm read or write data linearly at ~3x90 Mb/s... It's just silly. –  BarsMonster Dec 30 '10 at 0:04
show 6 more comments

try

echo 32768 > /sys/block/md0/md/stripe_cache_size

and check ;)

share|improve this answer
    
In previous answer we were testing different stripe_cache_sizes. Setting it to 32768 didn't improved anything. I am still at ~100Mb/sec writes even after upgrade (E8500 CPU, 8Gb ram and 8x1Tb array) –  BarsMonster Sep 10 '12 at 23:42
    
ABSURD! That instantly increased my write performance from ~70MB/s to ~360MB/s!?! Oh my god. W-T-F is this not default?!? –  stolsvik Jan 27 '13 at 1:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.