Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I've been experimenting with s3ql on Ubuntu 10.04, using it to mount Amazon S3 buckets. However, I'd really like it to mount them automatically. Does anyone know how to do that?


Solution:

Thanks to help from Nikratio of s3ql I'm finally able to mount S3 buckets automatically when the system boots. You'll definitely want to look at the manual, but here's the basics of how to do it!

The first step is to create an authinfo file. This file should be placed in a .s3ql directory within the home directory of the user who will be using it. The authinfo file contains login info allowing s3ql to mount buckets without prompting. Below is an example of what your authinfo file should look like. The first line contains your Amazon Security Credentials. The second contains the location and password for your bucket. You can add multiple bucket-lines to this file if needed, but I'm only using one in this example. At this point, the bucket password can be anything.

backend s3 machine any login YourAWSAccessKeyID password YourAWSSecretAccessKey
storage-url s3://mybucket password EncryptionPasswordOfYourChoosing

The encryption_password_of_your_choosing is used by S3QL to encrypt/decrypt all file writes/reads from S3.

The bucket name has to be unique across all of AWS: no two users can have the same bucket name, so its a good idea to log into Amazon Web Services and try different names until you find one that's available. One good naming scheme is to use the name of the domain+server you'll be accessing the bucket from (i.e. "staging.example.com" or "east.coast.01.example.com", or whatever is appropriate).

To create the file system, use the command:

mkfs.s3ql s3://mybucket

It will prompt you for your encryption password. This should be the same as the bucket password in the authinfo file.

Now that your file system is created, you can mount it using the command:

mount.s3ql s3://mybucket /mnt/s3/bucket

Of course, your bucket name and mount point will vary.

Now, if we want to mount this bucket automatically on boot, we need to add an upstart script to /etc/init. Fortunately, s3ql comes packaged with one, s3ql.conf.

I added "--allow-other" to the mount.s3ql command to allow users other than root to access the mounted bucket.

#
# This file can be placed in /etc/init. It defines an upstart job that
# takes care of mounting and unmounting an S3QL file system.
# 
description "S3QL Backup File System"
author      "Nikolaus Rath <Nikolaus@rath.org>"

start on (filesystem and net-device-up IFACE=eth0)
stop on runlevel [016]

env BUCKET="s3://mybucket"
env MOUNTPOINT="/mnt/s3/bucket"

expect stop

script
    # Redirect stdout and stderr into the system log
    DIR=$(mktemp -d)
    mkfifo "$DIR/LOG_FIFO"
    logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" &
    exec > "$DIR/LOG_FIFO"
    exec 2>&1
    rm -rf "$DIR"

    # Check and mount file system
    fsck.s3ql --batch "$BUCKET"
    exec mount.s3ql --upstart --allow-other "$BUCKET" "$MOUNTPOINT"
end script

pre-stop script
    umount.s3ql "$MOUNTPOINT"
end script

After adding this script, in theory you should be able to reboot and have your bucket automatically mounted, but this is were I ran into trouble. Mine wasn't being mounted.

My problem was caused by the fact that upstart was running the script as root, but I'd created the file system as another user. Once I copied the .s3ql directory from the home directory of the user I'd been logged-in as to /root, the problem was solved.

I hope this helps someone else out there. Although I haven't been using my mounted S3 bucket for long, I'm impressed with how my initial tests have gone.

Also, this answer was written about a week after the solution was found. I think I've covered everything, but if you find I've missed a step, let me know and I'll add it. You'll also want to read the manual, its really worth reading if you intend to use s3ql.

share|improve this question
1  
Can you specify how you're getting this script to run at boot and weather you have correctly ordered any init.d scripts to happen after the network is up. also remember that the network may not be available on a desktop until you've logged in, thus you may have to delay the script until you're logging in. –  Martin Owens -doctormo- Jan 8 '11 at 8:10
    
Thanks for responding. I'm running it on Ubuntu-server, with no desktop, so that's not an issue. I'm using "update-rc.d s3ql defaults", which should cause it to be triggered at boot. However, the fact that the network may not be available at the time it starts may be the problem. I've been experimenting with setting it with different boot/shutdown priorities, but I'm not sure I've done that right. It should probably be the last thing to run at boot and the first thing to stop at shutdown. –  ks78 Jan 8 '11 at 15:27
    
Here's an idea - setup a cronjob every X minutes that checks if it's running, and if it isn't start it. That should do the trick. But on the other hand, it won't let you set it's priority like you want to. –  jrg Jan 8 '11 at 18:00
    
I suppose setting up a cron job is a possibility, but I think there's better ways to start a process on boot and have it automatically stopped on shutdown. --I'm just trying to find one that will work for me. –  ks78 Jan 9 '11 at 0:46
    
While this Q/A was incredibly helpful for me, the format of the files has changed, at least in v1.12 that I'm using. The authinfo file is now formatted into sections, and there's an updated auto-mount script. Finally, with my blank 10.04 Ubuntu image from AWS EC2 I had to make sure that the fuse module was loaded. Since S3QL is the only piece of software that uses fuse on my server I decided to simply add the line sudo modprobe fuse to my S3QL init script. –  jefflunt Oct 14 '12 at 4:59
add comment

2 Answers 2

up vote 1 down vote accepted

With some help from Nikratio, I'm finally able to mount my S3 bucket automatically on boot. I've updated my question with the solution.

share|improve this answer
    
This really is the place for the answer. Convention would suggest that you should just put the answer here. Since the "accepted" answer is always listed first, it will always appear immediately after the question. :) –  jefflunt Oct 14 '12 at 5:08
add comment

Modern Ubuntu installations use the event driven upstart system for much of the boot up process, so you're best to move the start up to that. You'd have a script in /etc/init that would look something like this:

description     "S3FS"

start on (filesystem
        and net-device-up IFACE!=lo)
stop on runlevel[!2345]

console output
expect fork
respawn

script
    # Redirect stdout and stderr into the system log
    DIR=$(mktemp -d)
    mkfifo "$DIR/LOG_FIFO"
    logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" &
    exec > "$DIR/LOG_FIFO"
    exec 2>&1
    rm -rf "$DIR"

    modprobe fuse
    fsck.s3ql --batch s3://mybucket
    exec mount.s3ql --allow-other s3://mybucket /mnt/s3fs
end script

You'll have to do some testing work — I don't know how the output redirection for logging will behave, and you may need to set up a post-stop script to properly unmount the file system.

The upstart documentation is here. I haven't found a FUSE based package that uses it, yet, but I didn't do an exhaustive search.

An alternative may be to construct an fstab entry for the file system, so as to leave the mountall entries to take care of mounting it for you, but I don't know if there is a way to specify that a given FUSE file system is network dependent.

share|improve this answer
    
Thank you, jwm. I think upstart is what I need. I'm assuming anything that can be run from the command-line can be scripted. Do you know of any good upstart script examples? Your script is probably close to what I need, but it still doesn't start up properly. Could FUSE be a problem? –  ks78 Jan 10 '11 at 17:01
    
There are plenty of examples in /etc/init, and there might be some at the documentation site I linked to. I'd start debugging by stripping the log redirect parts. Also, I expect that upstart assumes that the command it exec's should be long running, so look for an existing script that runs something once to see how that works. –  John Morton Jan 12 '11 at 1:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.