2

I'd like to set up a software raid 10 array (4x2TB) for a workstation using a fresh install, but I'm finding non applicable/conflicting/old resources online. I'd like to get the community's advice on setting up such an array as there appears to be a plethora of possible configurations.

Specifically, I'll be using the workstation for image analysis (i.e., typical file sizes from several hundred MB-several GB), so if there are optimizations that can be made for such purposes that would be great.

Cheers

3 Answers 3

1

I don't remember exactly what resources I followed when setting up my RAID on my server, but I think this article was the main point of information. Some important things:

  • Use mdadm and not dmraid.
  • Use /dev/disk/by-id/ paths to point at the disks, instead of /dev/sda etc… It's easier to map them to the physical devices, in case you need to replace a disk or such.
  • Be patient. At first I thought my RAID would be ready after the 5 hours initial setup time. Then it took another day to rebuild itself and actually be ready. (4x2TB)
  • Use a separate disk/partition/RAID for primary OS installs. It's easier to manage everything if you keep the large data RAID separate from primary OS and its data. Much easier to recover a small OS disk, than rebuild a huge multi-terabyte array, if something goes bad on the OS side.
11
  • Not entirely, you still need to identify your drives in advance, preferably by serial number, so when one is reported failed it's easy to remove just that one. I'll take a label and copy the serial # to the cable end of the drive. The by-id name usually includes the serial number.
    – ppetraki
    Dec 14, 2012 at 14:21
  • @ppetraki Ok, fair enough. It would save you some time in mapping the physical disk visually to the name given. However, are you sure this actually works like this? I seems not to work for me when doing auto-assembling by array-UUID. It reports plain names in the array configuration (e.g. sda1) no matter what the original paths to disks were on the time of creation.
    – gertvdijk
    Dec 14, 2012 at 14:30
  • You don't need to use the by-id names in assembling the array, but does help you organize your plan, especially if you have tons of disks, they're just symlinks to SD devices. MD just scans for metadata, it doesn't care about disk names, neither does the admin tool, which is unfortunate as now I have to spend the extra time to lookup sda => scsi-SATA_TOSHIBA_THNS128_10US10BTT02Z, which is a lot more meaningful. It would be a great feature to add I just haven't got around to it.
    – ppetraki
    Dec 14, 2012 at 14:35
  • @dobey - so far, that's the best description for a RAID 10 setup I've seen. Thx. Pity that it's dated 2008. Dec 14, 2012 at 14:48
  • @Prophet60091 Regarding the mdadm commands, nothing has really changed. The Grub part is a bit outdated, though.
    – gertvdijk
    Dec 14, 2012 at 14:49
1

With RAID10 in the given situation I see only two variables candidate for optimization:

  • Chunk size

    Set it to something larger than the default of 512KiB to minimize the overhead for linear reads/writes of large files. You should try it on a small partition on your machine to see what gives the best performance, e.g. 1MB, 2MB, 5MB, 10MB...

  • Near vs Far layout

    comparable to RAID1+0 vs RAID0+1. Far is a bit faster as the performance for reading is more like RAID0. Yet a Near layout is the default because it has a slightly higher chance of surviving the unlikely event that all mirrored disks are broken (some probability math here). Some more visual idea of the differens is below, happily stolen from SLES mdadm documentation:

    Near looks like

    sda1 sdb1 sdc1 sdd1
      0    0    1    1
      2    2    3    3
      4    4    5    5
    

    Far looks like

    sda1 sdb1 sdc1 sdd1
      0    1    2    3
      4    5    6    7       
      . . .
      3    0    1    2
      7    4    5    6
    

Update about far vs near redundancy from the discussion in the comments. Suppose sda fails:

       near
sda1 sdb1 sdc1 sdd1
  -    0    1    1
  -    2    3    3
  -    4    5    5

then sdc or sdd can still fail, while in far:

        far
sda1 sdb1 sdc1 sdd1
  -    1    2    3
  -    5    6    7       
  . . .
  -    0    1    2
  -    4    5    6

now only sdc can fail, as a failed sdb drive make block 4 inaccessible and a failed sdd drive will make block 3 inaccessible.

Conclusion: chances of surviving a 2-disk failure are higher when using a near layout. (can someone do the math here for a quantitative number?)

7
  • thx for that link. I like the distinction made between "Complex vs. Nested RAID 10". That particular nuance was never clear to me. Dec 14, 2012 at 14:46
  • far doesn't have any better chance of survival ( data is on the same disk, just different location ), it is just optimized for reading rather than writing. There is also the offset layout, which is more of a balance between the two.
    – psusi
    Dec 14, 2012 at 15:28
  • 1
    @psusi No, that's exactly making the difference! Suppose sda fails. In near, then sdc or sdd can fail, but not sdb. In far, only sdc can fail because sdb can't fail (missing block 4) and sdd can't either (missing block 3).
    – gertvdijk
    Dec 15, 2012 at 0:01
  • 1
    No, both layouts can handle some double disk failures, just which ones differ. With far it is sdb and sdd or sda and sdc.
    – psusi
    Dec 15, 2012 at 1:21
  • 1
    Also worth noting is that the far layout can not be resized. I recently picked up 3 1 TB WD blue drives cheap and set them up in raid10,offset with 16M chunk size and it gets nearly 500 MB/s sequential read throughput. I only initialized part of the array ( with the -z switch ) to avoid the long wait for resyncing the whole thing, and can grow it as needed ( on the fly ).
    – psusi
    Jan 4, 2014 at 5:47
-3

Picking up some hotspares in advance would be a good idea. Also taking these notes into account.

Recommended storage scheme for home server? (LVM/JBOD/RAID 5...)

See footnote [1] in above link to see what happens with cheap storage when you need it the most.

This is all a moot point however until you profile how your the target application actually uses storage. You might find that parallelism is possible, so one block can be used for reading results, and one for writing them. This could be further abstracted behind a RAID0 (until the HBA reports QUEUE_FULL) with the results backed up via rsync.

It really depends, saying "I'm doing image analysis" without defining the workload or level of service just isn't enough; even if you did, that level of performance analysis is real work, I know it's something "I" wouldn't do in my spare time. My intentions are to get you thinking about your application to create your own solutions. Spindles are always the slowest part of your system, plan accordingly.

One idea if you wish to do the multi-array approach would be creating two RAID 1's, on separate controllers, and adding those MD devices to a LVM VG for management. Sure a RAID 10 is fast, but it's still one storage queue, now you have two, and with separate controllers there's no HBA queue sharing either.

Performance Notes:

Remember, SW RAID is no different than HW RAID, understand how it works or when it fails you might end up being more at risk as opposed to say spending your energies creating a regular backup strategy (rsync.net) instead. I've lost count of the number of users who've lost everything because they didn't read the manual and actually test the failure modes.

10
  • Yes, some amazing IO requirements are definitely in mind (image analysis). If I had the money, this would be purely an SSD affair. Also, it's my understanding that SW RAID is more flexible and cheaper that HW RAID...I've even read that it's faster, but that's neither here nor there for this particular question. Thx for the link though. Dec 14, 2012 at 14:38
  • In which case, asynchronous backup is cheaper, start with a RAID0 Rsync to replicate your work to a safe location. I recommend rsync.net, works well with open standards: rsync, duplicity, & deja-dup. You might want to have two arrays, on separate controllers. One that does all the reading of the original data, and one that serves as the destination of the processed result; nice for multi-threaded processing (pipeline), You also might want to consider using XFS or similar fs that excels at large files, the storage array is just one piece of the puzzle.
    – ppetraki
    Dec 14, 2012 at 14:47
  • interesting: so one RAID0 dedicated to reading, and another RAID0 dedicated to writing, all the while backing up the processed results. I'll have to explore this option as well. Thx. Dec 14, 2012 at 15:01
  • If you're doing batch processing, doing something as simple as dividing the list into 4 parts and spawning a pipeline for each segment for instant parallelism. You can get really cute here and chunk the list by total size and spawn a job for every 500GB for example.
    – ppetraki
    Dec 14, 2012 at 15:08
  • 1
    Several things are wrong with this answer: 1) lvm over raid10 is better than lvm over two raid1s, 2) you don't need to replace with the exact same model with or without partitions, 3) hardware or software raid does not matter; raid is no substitute for having regular backups!
    – psusi
    Dec 14, 2012 at 15:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .