8

I have a mdadm RAID-6 in my home server of 5x1Tb WD Green HDDs. Read speed is more than enough - 268 Mb/s in dd. But write speed is just 37.1 Mb/s. (Both tested via dd on 48Gb file, RAM size is 1Gb, block size used in testing is 8kb)

Could you please suggest why write speed is so low and is there any ways to improve it? CPU usage during writing is just 25% (i.e. half of 1 core of Opteron 165) No business critical data there & server is UPS-backed.

mdstat is:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sda1[0] sdd1[4] sde1[3] sdf1[2] sdb1[1]
      2929683456 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [5/5] [UUUUU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>

Any suggestions?

Things like writeback, barrier, nobh didn't helped. DD blocksize=1M, 8M didn't changed anything. It looks like mdadm physically reads sectors to calculate parity even if it does not matter... Is that correct?

Update: Speed degradation after altered stripe cache was actually because 1 HDD probably failed during testing, nice :-D

Resolved: After increasing stripe cache & switching to external bitmap, my speeds are 160 Mb/s writes, 260 Mb/s reads. :-D

2
  • Hey Bars.. What have you tweaked and after what change did the performance drop down to 120Mb/s? I am having a similar issue but my performance is way worse :( Any suggestions would be useful! Thanks!
    – user12968
    Mar 24, 2011 at 23:51
  • In my case I had a diskfail in the middle of optimization :-) Mar 25, 2011 at 4:16

2 Answers 2

9

Have you tried tuning /sys/block/mdX/md/stripe_cache_size?

According to this forum post (in Norwegian, sorry) "tuning this parameter is more essential the more disks and the faster system you have":

On my system I get the best performance using the value 8192. If I use the default value of 256 the write performance drops 66%.

Quoting his speed for comparison:

Disks: 8xSeagate 2TB LP (5900RPM) in mdadm RAID6 (-n 512) (stripe_size_cache=8192).

CPU: Intel X3430 (4x2.4GHz, 8GB DDR3 ECC RAM)

Speed: 387 MB/s sequential write, 704 MB/s sequential read, 669 random seeks per sec.

My home server has almost the same disks as you, using RAID 5:

Disks: 4x1.5TB WD Green in RAID 5 (stripe_size_cache=256 - the default)

CPU: Intel i7 920 (2.66 GHz, 6 GB RAM)

Speed: 60 MB/s sequential write, 138 MB/s sequential read (according to Bonnie++)

So it looks like sequential write performance is around 50% of read performance.

For what performance to expect, the Linux Raid Wiki says about RAID 5:

Reads are almost similar to RAID-0 reads, writes can be either rather expensive (requiring read-in prior to write, in order to be able to calculate the correct parity information, such as in database operations), or similar to RAID-1 writes (when larger sequential writes are performed, and parity can be calculated directly from the other blocks to be written).

And about RAID 6:

Read performance is similar to RAID-5 but write performance is worse.

12
  • Write speed dropped to 15-20 Mb/s, read speed down to 110 :-S This just does not make sence :-( Dec 29, 2010 at 23:06
  • Try some other values, see what happens. Not an expert on this, but I have WD Greens (4x1.5TB) in md RAID 5 with the same issue - rather slow write speed. My system is down at the moment, but I'll experiment at my end as well, and update here if I manage to improve it. Dec 29, 2010 at 23:13
  • BTW: Do you have AHCI enabled? Which file system - ext3, ext4? Dec 29, 2010 at 23:20
  • Not sure about AHCI, it's very hard to take a look at BIOS. ext4. Dec 29, 2010 at 23:30
  • Damn, stuck with 120Mb/s reading speed, can't get back to my 268 :-( This is so frustrating... I don't see why can't mdadm read or write data linearly at ~3x90 Mb/s... It's just silly. Dec 30, 2010 at 0:04
4

try

echo 32768 > /sys/block/md0/md/stripe_cache_size

and check ;)

4
  • In previous answer we were testing different stripe_cache_sizes. Setting it to 32768 didn't improved anything. I am still at ~100Mb/sec writes even after upgrade (E8500 CPU, 8Gb ram and 8x1Tb array) Sep 10, 2012 at 23:42
  • ABSURD! That instantly increased my write performance from ~70MB/s to ~360MB/s!?! Oh my god. W-T-F is this not default?!?
    – stolsvik
    Jan 27, 2013 at 1:19
  • Oh my word. This really works... From [================>....] recovery = 82.1% (1604609136/1953382144) finish=122.7min speed=47348K/sec to [================>....] recovery = 82.3% (1607712880/1953382144) finish=56.2min speed=102451K/sec
    – Lmwangi
    Aug 1, 2015 at 14:55
  • In my testing, the value of 8192 seems sufficient. Despite throwing everything I had to test with at an array, I couldn't get stripe_cache_active to spike higher than the low 7000s. Nov 11, 2015 at 3:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .