Can I cache an mdadm raid with an ssd?

Question

I have a 1TB array set up in raid 1 using mdadm on Ubuntu 8, to store mission-critical files. I'd like to speed up read/write, especially for large files by caching to an SSD. Is this possible? Does mdadm contain a facility to cache to another drive?

Alternatively, is there something I should do to assign ordinary RAM to caching? There is 4GB RAM and files to be written are rarely larger than 1GB so I'm thinking writes should be quite quick just with RAM caching alone.

Any help will be greatly appreciated.

Answer 1

Does mdadm contain a facility to cache to another drive?

No, the Linux software raid (managed by mdadm) is purely for creating a set of disks for redundancy purposes.¹

I know of two projects that will let you do this: dm-cache and flashcache (both relatively experimental). It enables you to use your SSD as cache (read and write) for your slower hard drives or any other block device such as an md device. Both offer huge performance gain and are very tweakable to find balance in durability and performance.

Unfortunately, this not yet packaged or included in the kernel as of this time. Therefore it's not a very straightforward task to do in Ubuntu. Feel free to get your hands dirty, but remember it's a bit of experimental software and hard to get support on.

Related questions:

• How do I install and use flashcache/bcache to cache HDD to SSD?
• raid advice with SSD and two HDD

¹ _{However, you can hint the kernel to use one of the drives for reading primarily, to get some improvement in reading performance, but this is not quite the same as a real SSD cache. See this answer by Gilles on how to do this.}

---

is there something I should do to assign ordinary RAM to caching?

This is already enabled in every Linux kernel! You can see the amount of cached pages by using free:

free -m
             total       used       free     shared    buffers     cached
Mem:         24047      17703       6343          0       6492       3550
-/+ buffers/cache:       7660      16386
Swap:         3811          0       3811

Here you see that it has now 3550 MB of pages (file system level) in cache, and 6492 MB of pages (block device level) on my machine. Reads to one of those will not result in a read from disk.

To demonstrate block-level caches this, try this command:

hdparm -Tt /dev/md127  # replace with your device - try both disks and md devices!

/dev/md127:
 Timing cached reads:   8624 MB in  2.00 seconds = 4313.50 MB/sec
 Timing buffered disk reads: 282 MB in  2.93 seconds =  96.31 MB/sec

The first result is clearly coming from caches in memory, whereas the second is read directly from disk.

To demonstrate the file system based caching using dd:

# first time
dd if=/home/gert/bigfile.img of=/dev/null 
5927206912 bytes (5.9 GB) copied, 25.9253 s, 229 MB/s

# second time
dd if=/home/gert/bigfile.img of=/dev/null 
5927206912 bytes (5.9 GB) copied, 4.91444 s, 1.2 GB/s

# third time, with file system cache disabled using iflag=direct
# yields results similar to first one
# Try using a bigger blocksize (bs=10M for example) if you're seeing very low speeds
dd if=/home/gert/bigfile.img of=/dev/null iflag=direct

Answer 2

A somewhat similar solution for RAID4/5/6: https://www.kernel.org/doc/Documentation/md/raid5-cache.txt

It's an option to use an additional drive to your RAID4/5/6 configuration solely for caching purposes. There are also modes to either increase performance (read or write) or reliability.