5

I have a new machine with one 128GB SSD and two 1TB HDD. On the SSD is the OS and my initial thought was to put the two HDD in RAID 1 for user data.

After some more thought I came up with two other setups and now I'm in doubt :) Can someone advise what would be the best setup?

1: single SSD and HDD in RAID 1 (original thought)

2: Create 2 partitions on the HDD (128GB and 872GB). Put the two 872GB in RAID 1 and create another RAID 1 with the SSD and one 128GB HDD partition.

3: Create 2 partitions on the HDD (750/250), put the 705GB in RAID 1 and use the 2 250GB as backup and make automatic snapshots of the SSD to (one of) these partitions.

I think the 2 main questions are:

Is it advisable to create a raid array with only part of a drive and actively use the other part of that drive or should you always use the full disk?

Is it advisable to create a raid 1 array with a SSD and HDD or will that blow the whole speed advantage of the SSD?

5 Answers 5

4

I would like to add a different recommendation to the pool of possible solutions. I would recommend you to base your setup on btrfs' subvolume and snapshot abilities in combination with a btrbk cronjob.

Setting this up is not necessarily trivial but largely depends on your skill set and previous experience. There is a lot of literature on the net that will help you. In the end you are rewarded with a very flexible and fast way to backup your SSD regularly with a lot of options to create a solution perfect for your needs.

Note of caution: Any form of raid can and should never replace regular backups. (Luckily btrbk can easily be extended to external drives or ssh reachable host, see its manual)

General setup

The general ideas of my proposal is to use the SSD as your btrfs system drive that contains your root and related subvolumes and the two HDDs in a btrfs raid1 as your data and backup drive.

Together with btrbk this will allow you to perform automated incremental backups of your system SSD to your backup HDDs. And since the HDDs are set up as a mirror, all your backup will be kept mirrored as well.

Furthermore btrfs ability to send and receive subvolumes (which is what btrbk utilizes to make backups) allow you to freely move your data and backups between your system and data drive. This will allow you to change what data is stored on the fast SSD, while always maintaining versioned and mirrored backups of all your data.

Setup btrfs

To get started you will either need to reinstall Ubuntu onto the SSD and selecting btrfs as your root file system or to convert your existing installations file system to btrfs. Both ways are described on the Ubuntu's community help page about btrfs, which is a good read in general, if you are just getting start with btrfs on Ubuntu.

Next you need to turn the HDDs into a btrfs raid1, with the following command, where /dev/sdx and /dev/sdy are the two drives (All data on these drives will be lost!):

mkfs.btrfs -d raid1 /dev/sdx /dev/sdy

If you are new to snapshots or btrfs, I would recommend the this moment to familiarize yourself with the difference between folders, subvolumes and snapshots and try out some of the commands, before any actual data is written to your raid1.

There are many ways you can organize your data and you can find some examples in the btrfs kernel wiki's sysadmin guide.

One way to do it is to mount your btrfs root (subvolid 0 or 5) somewhere and use that to manage your subvolumes and snapshot and furthermore store all of the data in appropriate subvolumes, which you mount to convenient locations in your file system. That way you can snapshot, move, recover and replace any data at will.

For your concrete example, that could mean the following (all command should be run as root/with sudo):

  • Mount your system btrfs root (subvolid=0) to /btrfs/system
  • Mount your data btrfs raid1 root (subvolid=0) to /btrfs/data

Instead of mounting these volumes by hand, add them to your fstab (/etc/fstab) before mounting, so they are mounted at boot as well. I would recommend to mount them by their UUID, which you can retrieve by running sudo btrfs filesystem show.

UUID=<UUID of system> /btrfs/system btrfs defaults,subvolid=0 0 0
UUID=<UUID of data> /btrfs/data btrfs defaults,subvolid=0 0 0

Now mount them with:

sudo mkdir /btrfs
sudo mkdir /btrfs/data
sudo mount /btrfs/data
sudo mkdir /btrfs/system
sudo mount /btrfs/system

Now you can any additional subvolumes you might want to each of the btrfs filesystems. Ubuntu normally creates a subvolume for your root / (subvol=@) and home directory /home (subvol=@home) by default. It is common to turn /var or /tmp into their own subvolumes or to create application specific subvolumes, e.g. for /var/www/.

Personally I prefer to keep all my subvolumes at the btrfs root and them mount them to their specific locations using mount and fstab entries.

For example, to create a subvolume for your music collection on the HDD raid1, I would do the following:

btrfs subvolume create /btrfs/data/@music

I would then mount it with the following fstab entry to /music:

UUID=<UUID of data> /music btrfs defaults,subvol=@music 0 0

Setup of btrbk

Secondly, you will have to set up btrbk for the subvolumes you want to snapshot and backup onto the HDD raid.

As a simple example on how to backup @ and @home regularly and keeping regular spaced history of your backup you could write the following to /etc/btrbk.conf:

# The long timestamp recommended for more then one snapshot a day
timestamp_format        long

# Set time spacing of snapshots kept on SSD
snapshot_preserve_min      2d
snapshot_preserve          7d 4w 3m

# Set time spacing of snapshots kept on HDD raid
target_preserve_min        no
target_preserve            8w *m

snapshot_dir               /btrfs/system/snapshots

volume /btrfs/system
  subvolume @
    target send-receive /btrfs/data/backup/

  subvolume @home
    target send-receive /btrfs/data/backup/

Please read the btrbk documentation for any of the details. It will also explain how to recover your data from a snapshot.

Lastly you will have to add btrbk to your crontab with sudo crontab -e. E.g. to run your btrbk snapshots and backup every day at noon, add the following line:

0 12 * * * /path/of/btrbk run

Other considerations

Swap

While in general there is less and less need for swap space in modern personal computer systems that have at least 8 GB of RAM, there are still use cases where it can help you out, especially when located on a SSD, where the performance hit of swapping is not as noticeable. It is therefore still generally recommended to set up a swap file or partition.

That being said, btrfs does not support swap files. That means, you will have to allocate some of your SSD space into a separate swap partition if you want to be able to use swap on your system at all.

btrfs SSD detection

Btrfs automatically detects if a file system mounted is located on a SSD and enables wear-leveling in that case.

This however is not necessary anymore, as modern SSDs automatically wear-level themselves, while at the same time causing issues with the fragmentation of free space. I would therefore personally advice you to mount your SSD with the nossd option.

More details can be found in the btrfs kernel wiki.

File system compression

Btrfs supports transparent file compression. By adding the compress option to your mount flags, it will be enable for all new files written.

For example, to enable compression for the @music subvolume I used as an example earlier, I would change my fstab entry to:

UUID=<UUID of data> /music btrfs defaults,compress,subvol=@music 0 0

To apply this change, do not forget to remount (unmount and mount again) the effected subvolume.

With most recent computers the overhead of compressing files before it is being writing and after it has been read is often negligible. Writing a large but well compressible file to a slow disk might even be faster with compression enabled.

If you are worried about speed, you can also use the faster but less efficient LZO compression with compress=lzo.

More details once again can be found in the btrfs kernel wiki.

Location of home folder

Ubuntu puts your home folder into the @home subvolume on the system drive by default. If you move home to your HDD mirror is up to your personal preferences. You can also keep it on the SSD and include it into your btrbk backup as I showed above.

Time spacing of backups

While my example set the same retention times for all subvolumes, btrbk allows you to set this time for each of them individually.

You could also run btrbk with different configs (see -c option) at different intervals to gain even more control over when what subvolume is snapshotted amd/or backuped.

As snapshots are quickly created, thanks to btrfs copy-on-write nature, you could even get btrbk to create a snapshot each hour, but only transfer them to the backup disk once a day, for folders that are changed often, like your home folder.

Detecting bit rot

To make use of btrfs bit rot/data corruption detection (and automatic repair in case of your raid1), you should make sure to run a btrfs scrub at regular intervals, e.g. as a cron job, with:

btrfs scrub start /btrfs/system
btrfs scrub start /btrfs/data
2
  • 1
    I think I have never seen a better answer from a new user. What links would you like to add? I'll add them for you,
    – fosslinux
    Jul 24, 2017 at 1:17
  • @ubashu Thank you for your praise and offer of support. Answering this question luckily awarded me the needed reputation and I went ahead and added the missing links myself. Jul 24, 2017 at 12:20
3

The first option is not completely stupid. "Hybrid RAID1" does work if configured properly. I have ran this setup myself until my SSD started acting weird and moved out of the box.

See these articles for guidance:

http://tansi.info/hybrid/

http://www.vinsec.net/2012/05/hybrid-hdd-ssd-raid1.html

https://superuser.com/questions/293144/combining-ssd-and-hard-disk-in-software-raid1

2

Is it advisable to create a raid array with only part of a drive and actively use the other part of that drive or should you always use the full disk?

The advice is always to use as much similar devices as you can. I'm not sure how much of that advice is superstition though, but lets look at this first:

Is it advisable to create a raid 1 array with a SSD and HDD or will that blow the whole speed advantage of the SSD?

No. Don't do that, you will blow the speed. Apart from the fact that they are different devices etc, the one thing I am quite sure of is that your speeds will drop, as you are going to be working at the speed of the slowest disk -> your ssd-hdd combo will be way slower then your SSD :

You should NOT do that to your SSD. Your first option is the best one. Just give yourself safe userspace, and speedy system disk. Backup that disk. (remember, RAID is not a backup strategy)

3
  • 2
    The advice is always to use as much similar devices as you can. -- The more the drives are identical and used identically, it tends to fail on identical times too. My suggestion is to find similar but not identical drives of the same size, yet different manufacturing date or brand.
    – gertvdijk
    Nov 15, 2012 at 10:06
  • I agree completely.
    – Nanne
    Nov 15, 2012 at 10:08
  • 1
    If you set the HDD to write-mostly then writes will be as slow as the HDD, but reads will not use the HDD at all and you'll get SSD read speeds. This could be a happy medium depending on the use case; I know my workstation sees far more reads than writes, and I'm using exactly this setup (SSD mirrored with write-mostly HDD). The performance is pretty similar under average workloads and I know that the SSD dying won't make me lose hours of work trying to get my machine running again. Just pop another in and let it rebuild.
    – cdhowie
    Sep 6, 2019 at 1:10
2

First question

For performance reasons, don't do this. It will degrade performance when I/O is performed on the other partition. However, if performance is not of your concern, but redundancy is, you can consider this.

Second question

No, don't do this. The chain is as strong as the weakest link here. Your array will perform more like a HDD and you'll waste an expensive SSD.

Instead, use...

dm-cache or flashcache (quite advanced and experimental). It enables you to use your SSD as cache (read and write) for your slower hard drives. Huge performance gain and tweakable to find balance in durability and performance.

See also: How do I install and use flashcache/bcache to cache HDD to SSD?

6
  • I'm not familiar with those cache solutions, but wouldn't you (just like swap) want to minimize those kind of things on your SSD, as they would wear out your device rather sooner then you'd want?
    – Nanne
    Nov 15, 2012 at 10:05
  • @Nanne No, you should just use your SSD for its speed. If you really worry about wear on your drive then disable write caching and use read caching only. I also suggest you to read more about write endurance and why it will probably not affect you here.
    – gertvdijk
    Nov 15, 2012 at 10:11
  • I agree you should use the SSD for speed, that's obvious. I'm just not knowledgable about the inner workings of the SSD enough to know if the worries people have are indeed valid: are they 'bs' for the whole "no swap!" thing as well? About read caching: that wouldn't really matter for the device used for storing the cache, would it?
    – Nanne
    Nov 15, 2012 at 10:19
  • @Nanne Read operations are very light for an SSD. You should indeed avoid unnecessary heavy write operations, but stay realistic. In this Intel PDF document in section 2.3 mentions 15TB as total write endurance for the 120GB drive. You may decide to turn off the flashcache for a while when transferring the bulk of stuff when starting.
    – gertvdijk
    Nov 15, 2012 at 10:28
  • 1
    What I meant was that using the disk as read-cache wouldn't neccessairily mean you would write less, would it? The cached data still needs to be written to the SSD, so the fact that you are caching only "read" stuff from the HDD, doesn't really matter, does it? (to be clear: not saying you're not right, just trying to wrap my head around it :) )
    – Nanne
    Nov 15, 2012 at 10:33
1
  1. is probably best.

You could put everything in a BTRFS RAID-1 array, you would get 1064 GB out of it (as per https://carfax.org.uk/btrfs-usage/ ) and BTRFS optimizes reads (not writes) s.t. they come from the SSD when data is available there, but it is more likely that you know exactly what 128 GB you need to access faster (eg system partition) and you can make it its own partition to avoid the random allocation.

I would avoid having multiple partitions per hard drive that are each part of multiple RAIDs, it could make access times unpredictible when your filesystem assumes they should be balanced equally but one is currently twice as busy. Splitting the SSD into a single partition and one that's part of the RAID-1 should be fine, but there is very little benefit with that little space.

1
  • "... BTRFS optimizes reads (not writes) s.t. they come from the SSD when data is available there... " This appears not to be the case. It is listed in the BtrFS Project Ideas, but it doesn't appear anyone has taken it up. On the other hand, it appears you can do it with mdadm, setting the HDD to "write mostly". Also, this will be/is a part of BcacheFS.
    – Diagon
    Dec 8, 2020 at 4:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .