Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

Are there any guidelines for which storage scheme(s) makes most sense for a multiple-disk home server?

I am assuming a separate boot/OS disk (so bootability is not a concern, this is for data storage only) and 4-6 storage disks of 1-2 TB each, for a total storage capacity in the range 4-12 TB.

The file system is ext4, I expect there will be only one big partition spanning all disks.

As far as I can tell, the alternatives are

individual disks

  • pros: works with any combination of disk sizes; losing a disk loses only the data on that disk; no need for volume management.
  • cons: data management is clumsy when logical units (like a "movies" folder) are larger than the capacity of any single drive.

JBOD span

  • pros: can merge disks of any size.
  • cons: losing a disk loses all data on all disks

LVM

  • pros: can merge disks of any size; relatively simple to add and remove disks.
  • cons: losing a disk loses all data on all disks

RAID 0

  • pros: speed
  • cons: losing one drive loses all data; disks must be same size

RAID 5

  • pros: data survives losing one disk
  • cons: gives up one disk worth of capacity; disks must be same size

RAID 6

  • pros: data survives losing two disks
  • cons: gives up two disks worth of capacity; disks must be same size

I'm primarily considering either LVM or JBOD span simply because it will let me reuse older, smaller-capacity disks when I upgrade the system. The runner-up is RAID 0 for speed.

I'm planning on having full backups to a separate system, so I expect the extra redundancy from RAID levels 5 or 6 won't be important.

Is this a fair representation of the alternatives? Are there other considerations or alternatives I have missed? And what would you recommend?

share|improve this question
    
This may be a better post for Server Fault –  Marco Ceppi Dec 26 '10 at 13:09
1  
I'll try there if I don't get a response here. But setting up a home server with a few extra disks is becoming relatively mainstream, and a far cry from managing enterprise servers, so I believe the question more naturally belongs here. Plus the next person considering an Ubuntu-based home server is more likely to check this site for answers. –  j-g-faustus Dec 26 '10 at 13:49
    
Another alternative you missed is using a multi-disk btrfs filesystem. –  poolie Oct 17 '11 at 23:06
    
One thing to consider, in thinking about redundancy, is how long it will take you to recover. If the other backup is off site connected over (eg) a 20Mbps line, restoring everything would take ten weeks. If it's local over GBE, it's still nearly two days. –  poolie Oct 17 '11 at 23:13
    
@poolie At the time I wrote this, my impression was that btrfs simply wasn't stable enough yet. But the multi-disk feature seems nice, I guess I'll consider switching when btrfs becomes the official Ubuntu file system. Another thing I left out was RAID 10, which is what I ended up with on one of the machines. Recovery time from backup is currently ~8 hours for 3.5 TB data, which is fine. But anything off site over internet would be too slow to be practical. –  j-g-faustus Oct 18 '11 at 2:03

4 Answers 4

up vote 4 down vote accepted

Like you I'm going through a rationalisation process with the disks in my home server. I too have a mix of disk sizes resulting from the organic growth of the JBOD setup I have.

I am taking the LVM route for the following reasons.

  1. Its the simplest
  2. It allows me to reuse the disks I already have in the server
  3. I have a complete backup of all the data that I am confident I can restore from
  4. I am not concerned about the recovery time in the event of a disk failure

For me the clinching factors are #3 & #4.

share|improve this answer
    
I'll buy that. I think the flexibility of LVM wins when taking your 3 & 4 into account. –  j-g-faustus Jan 2 '11 at 12:01
    
One other advantage of LVM over JBOD or individual disks: you can stripe across the disks and possibly get better IO speed. –  poolie Oct 17 '11 at 23:06

well on raid systems not the disks must have the same size...

just the partitions you want to add to the raid, need to have the same size to create a raid...

the strengths of lvm are, that you can easily grow your virtual disk by adding more partitions to it. and you have a snapshotting feature!

you can also combine lvm with raid... so that you have data security and the flexibility of lvm :)

share|improve this answer
    
It's true that just the RAID partitions need to be the same size, but not sure what to do with the remaining space. Ex. three disks 1.0TB, 1.5TB and 2.0TB could be RAID 5 using 3x1.0TB partitions, but that would leave 1.5TB unused space. I could stuff all of it into LVM (so the LVM runs on one RAID and two non-RAID volumes), but I can't immediately see that "partial RAID" has any advantages. Agree that RAID 5/6 (using similar-sized disks) + LVM may be the best option for a resilient single system. I like the LVM flexibility and snapshotting, so I'm leaning towards LVM. Thanks for input! –  j-g-faustus Dec 26 '10 at 18:59
    
gagme.com/greg/linux/raid-lvm.php –  JMW Dec 26 '10 at 21:01

You can stack block devices in Linux and mixin the value of both Software RAID and LVM which should address all your needs. This can all be accomplished from the non-gui installer.

  • Use a single partition that spans the 99% of the disk [1]
  • Create an MD RAID5 (preferably RAID6) with at least one hot spare
  • Initialize the MD array
  • Create an LVM VG
  • Add each MD device as a Physical Volume to the new VG [2]
  • Proceed to add swap and root logical volumes to VG
  • Format root with choice of filesystem (default is ext4)
  • Continue with installation

[1] I encountered a very nasty fault once on SATA disks that had lots of bad blocks. After using the vendor tool to reconstitute the disk. My once identical set of disks was now unique, the bad drive now had a few blocks less than before the low level format had begun, which of course ruined my partition table and prevented the drive from rejoined the MD RAID set.

Hard drives usually have a "free list" of backup blocks used for just an occasion. My theory is that that list must have been exhausted, and since this wasn't an enterprise disk, instead of failing safe and allowing me the opportunity to send it off for data recovery, it decided to truncate my data.

[2] Never deploy LVM without a fault tolerant backing store. LVM doesn't excel at disaster recovery, you're just asking for heartache and if you get it wrong, data loss. The only time it makes sense is if the VG group is confined to a single disk, like an external usb disk or perhaps an external eSATA RAID. The point is try to deploy your VG around backing stores that can hot plugged as a single unit, or as a virtual single unit which is demonstrated in the MD example above.

share|improve this answer

What about http://zfsonlinux.org/

It has the notion of disk pools that you can attach detach drives, I don't know if its production ready but still worth checking out.

share|improve this answer
    
Could be interesting in a few years :) Looking for a stable, low-hassle system myself - otherwise I'd probably go with btrfs - but looks interesting, and could perhaps be relevant for the more experimental? –  j-g-faustus Dec 27 '10 at 12:46
    
BTW, there is also Greyhole, which manages a pool of drives somewhat like Windows Home Server. –  j-g-faustus Dec 27 '10 at 12:56
    
Cool thanks for sharing –  narkisr Dec 27 '10 at 13:38
    
Not ready. It's really amazing how short people's memories are. LVM + MD has been around for about the last decade for this sort of provisioning. If it wasn't MD it was hardware raid. Point is IS has survived just fine without the latest whiz bang filesystem by adhering to this practice called regular backups. Eventually btrfs will be ready to integrate everything LVM + MD has to offer plus some other nifty features but until that time. The best you can do is contribute to the testing effort. I wouldn't recommend btrfs for production either. –  ppetraki Nov 10 '11 at 19:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.