10

I've just read this in another post about improving RAID5/6 write speeds:

After increasing stripe cache & switching to external bitmap, my speeds are 160 Mb/s writes, 260 Mb/s reads. :-D

I've already found out how to increase the stripe cache and this worked pretty well but I'd like to know more about an external bitmap. I have an incredibly fast (540MB/s) RAID0 SSD that would do well if a bitmap does what I think it does but I'm still very unsure. I've only known about them as long as I've known this post.

A few questions:

  • What is a bitmap (in terms of mdadm)?
  • What are the advantages of an internal bitmap (over external)?
  • What are the advantages of an external bitmap (over internal)?
  • How do I switch between the two?

I should add that while this is a I'm-bored-let's-break-something thread, I do value the data stored on the RAID array. If doing this is going to put data at significant risk, please let me know.

0

2 Answers 2

12

What's a bitmap:

A mdadm bitmap, also called a "write intent bitmap", is a mechanism to speed up RAID rebuilds after an unclean shutdown or after removing and re-adding a disk.

With a bitmap, writing data to the RAID goes like this:

  • Update bitmap: Mark the RAID chunks you are about to write to as dirty.
  • Write the data to the RAID.
  • Update bitmap: Mark the RAID chunks that were just written as clean.

The advantage of a bitmap is that if the system goes down in the middle of a write, the rebuild needs to check only the chunks marked as dirty, rather than the whole multi-TB RAID. This can speed up the rebuild process from taking several hours to completing in just a few seconds.

The drawback is lower write performance under normal use (outside rebuilds), since mdadm does additional disk access to update the bitmap.

External vs internal:

  • external: Stored as a file on a disk outside the RAID. The advantage over an internal bitmap is better write performance during normal use (outside rebuilds).
  • internal: Stored as RAID metadata. The advantage over an external bitmap is that you don't need a non-RAID disk and you save a bit on configuration (the path to the bitmap).

According to the mdadm man page:

Note:  external bitmaps are only known to work on ext2 and ext3.
       Storing bitmap files on other filesystems may result in  serious
       problems.

According to a post by Neil Brown, the mdadm author, external bitmaps should work on ext4 too:

I haven't looked inside ext4 but I am fairly confident that external bitmaps 
will work properly.

HOWTO:

Bitmaps are added and removed using mdadm --grow --bitmap=XXX ..., where the XXX is one of:

  • --bitmap=internal: Create an internal bitmap.
  • --bitmap=/var/my_bitmap.bin: Create an external bitmap at the specified path. The path must reside outside the RAID. A bitmap=... parameter must be added to the ARRAY entry in /etc/mdadm/mdadm.conf, and the --bitmap=... parameter must be passed if you are assembling the RAID from the command line.
  • --bitmap=none: Remove/disable any bitmaps.

References:

Comments:

IMO, bitmaps are perhaps primarily of interest for RAID levels 5 and 6, since these have the slowest rebuilds.

I switched from RAID 5 to RAID 10 myself; the rebuilds are so much faster that I don't feel the need for a bitmap, and RAID 10 seems to require far fewer rebuilds in the first place.

My RAID 5 setup used to drop a disk something like once a month, causing 12-14 hour rebuilds. The RAID 10 has only dropped a disk once in half a year, rebuilt in less than an hour.

I don't know if the frequent disk drops I experienced was caused by something other than the RAID level, but the RAID 10 has been far more stable and rebuild speed isn't much of a concern anymore.

1
  • 1
    For larger drives or write-intensive drives, you should also look at the --bitmap-chunk=NNN argument. This can often be as small as 4KB to 128KB by default which causes lots of bitmap updates. Increasing this value to 4096 (4MB) or 16384 (16MB) or 65536 (64MB) can help out with write performance, while still giving you fairly fast rebuilds. Use: "mdadm -G --bitmap=none" to remove existing bitmaps, then re-add with "mdadm -G --bitmap=internal --bitmap-chunk=N /dev/mdX"
    – tgharold
    Mar 27, 2013 at 12:10
1

I have been using md RAID 5 for a while now, and not had any disk dropouts. I was interested to see what the addition of an external bitmap storage would gain me.

My current setup includes a boot disk and then 4 x 3TB disks as RAID 5. Last night I added another 3TB disk and after 20 odd hours it's 70% thru reshaping. Since reading this post, I've decided that after it's finished reshaping I will definitely add the external bitmap to the boot disk (1TB).

I have md RAID 5 base layer, LVM2 over that and some ISCS targets to some of the LVM2 partitions with various virtual machines being tested on the ISCSI.

I've done a bunch of benchmark images, posted here if you're interested.

1
  • And how much corrupt data do you have? There is good reason why people don't use RAID5 anymore on big drives.
    – JamesRyan
    Mar 17, 2014 at 6:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .