How to create a s3ql file system and mount it automatically on boot?

Question

I've been experimenting with s3ql on Ubuntu 10.04, using it to mount Amazon S3 buckets. However, I'd really like it to mount them automatically. Does anyone know how to do that?

---

Solution:

Thanks to help from Nikratio of s3ql I'm finally able to mount S3 buckets automatically when the system boots. You'll definitely want to look at the manual, but here's the basics of how to do it!

The first step is to create an authinfo file. This file should be placed in a .s3ql directory within the home directory of the user who will be using it. The authinfo file contains login info allowing s3ql to mount buckets without prompting. Below is an example of what your authinfo file should look like. The first line contains your Amazon Security Credentials. The second contains the location and password for your bucket. You can add multiple bucket-lines to this file if needed, but I'm only using one in this example. At this point, the bucket password can be anything.

backend s3 machine any login YourAWSAccessKeyID password YourAWSSecretAccessKey
storage-url s3://mybucket password EncryptionPasswordOfYourChoosing

The encryption_password_of_your_choosing is used by S3QL to encrypt/decrypt all file writes/reads from S3.

The bucket name has to be unique across all of AWS: no two users can have the same bucket name, so its a good idea to log into Amazon Web Services and try different names until you find one that's available. One good naming scheme is to use the name of the domain+server you'll be accessing the bucket from (i.e. "staging.example.com" or "east.coast.01.example.com", or whatever is appropriate).

To create the file system, use the command:

mkfs.s3ql s3://mybucket

It will prompt you for your encryption password. This should be the same as the bucket password in the authinfo file.

Now that your file system is created, you can mount it using the command:

mount.s3ql s3://mybucket /mnt/s3/bucket

Of course, your bucket name and mount point will vary.

Now, if we want to mount this bucket automatically on boot, we need to add an upstart script to /etc/init. Fortunately, s3ql comes packaged with one, s3ql.conf.

I added "--allow-other" to the mount.s3ql command to allow users other than root to access the mounted bucket.

#
# This file can be placed in /etc/init. It defines an upstart job that
# takes care of mounting and unmounting an S3QL file system.
# 
description "S3QL Backup File System"
author      "Nikolaus Rath <[email protected]>"

start on (filesystem and net-device-up IFACE=eth0)
stop on runlevel [016]

env BUCKET="s3://mybucket"
env MOUNTPOINT="/mnt/s3/bucket"

expect stop

script
    # Redirect stdout and stderr into the system log
    DIR=$(mktemp -d)
    mkfifo "$DIR/LOG_FIFO"
    logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" &
    exec > "$DIR/LOG_FIFO"
    exec 2>&1
    rm -rf "$DIR"

    # Check and mount file system
    fsck.s3ql --batch "$BUCKET"
    exec mount.s3ql --upstart --allow-other "$BUCKET" "$MOUNTPOINT"
end script

pre-stop script
    umount.s3ql "$MOUNTPOINT"
end script

After adding this script, in theory you should be able to reboot and have your bucket automatically mounted, but this is were I ran into trouble. Mine wasn't being mounted.

My problem was caused by the fact that upstart was running the script as root, but I'd created the file system as another user. Once I copied the .s3ql directory from the home directory of the user I'd been logged-in as to /root, the problem was solved.

I hope this helps someone else out there. Although I haven't been using my mounted S3 bucket for long, I'm impressed with how my initial tests have gone.

Also, this answer was written about a week after the solution was found. I think I've covered everything, but if you find I've missed a step, let me know and I'll add it. You'll also want to read the manual, its really worth reading if you intend to use s3ql.

Answer 1

As I have not found a solution to this question using systemd (Ubuntu 20.04, s3ql 3.8.0) easily I will post my solution here:

[Unit]
Description=mount s3ql filesystem
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=all
ExecStart=mount.s3ql --systemd --fg <other options> <storage url> <mount point>
ExecStop=umount.s3ql <mount point>
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target

For very large filesystems it might be necessary to increase TimeoutStopSec or set it to infinity (which I personal wanted to avoid).

Add this to the correct systemd dir (e.g. /lib/systemd/system/s3ql.service) and systemctl daemon-reload && systemctl start s3ql (while the fs is unmounted and uncorrupted).

Answer 2

Modern Ubuntu installations use the event driven upstart system for much of the boot up process, so you're best to move the start up to that. You'd have a script in /etc/init that would look something like this:

description     "S3FS"

start on (filesystem
        and net-device-up IFACE!=lo)
stop on runlevel[!2345]

console output
expect fork
respawn

script
    # Redirect stdout and stderr into the system log
    DIR=$(mktemp -d)
    mkfifo "$DIR/LOG_FIFO"
    logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" &
    exec > "$DIR/LOG_FIFO"
    exec 2>&1
    rm -rf "$DIR"

    modprobe fuse
    fsck.s3ql --batch s3://mybucket
    exec mount.s3ql --allow-other s3://mybucket /mnt/s3fs
end script

You'll have to do some testing work — I don't know how the output redirection for logging will behave, and you may need to set up a post-stop script to properly unmount the file system.

The upstart documentation is here. I haven't found a FUSE based package that uses it, yet, but I didn't do an exhaustive search.

An alternative may be to construct an fstab entry for the file system, so as to leave the mountall entries to take care of mounting it for you, but I don't know if there is a way to specify that a given FUSE file system is network dependent.

Answer 3

With some help from Nikratio, I'm finally able to mount my S3 bucket automatically on boot. I've updated my question with the solution.