How to create a s3ql file system and mount it automatically on boot?

Question

I've been experimenting with s3ql on Ubuntu 10.04, using it to mount Amazon S3 buckets. However, I'd really like it to mount them automatically. Does anyone know how to do that?

---

Solution:

Thanks to help from Nikratio of s3ql I'm finally able to mount S3 buckets automatically when the system boots. You'll definitely want to look at the manual, but here's the basics of how to do it!

The first step is to create an authinfo file. This file should be placed in a .s3ql directory within the home directory of the user who will be using it. The authinfo file contains login info allowing s3ql to mount buckets without prompting. Below is an example of what your authinfo file should look like. The first line contains your Amazon Security Credentials. The second contains the location and password for your bucket. You can add multiple bucket-lines to this file if needed, but I'm only using one in this example. At this point, the bucket password can be anything.

backend s3 machine any login ASDFGHJKL password ZXCVBNM
storage-url s3://mybucket password mypassword

The bucket name has to be unique. No two users can have the same bucket name, so its a good idea to log into Amazon Web Services and try different names until you find one that's available. Once you've found one, you'll need to delete it, since s3ql will re-create it when it creates your s3ql filesystem. If the bucket already exists, you'll receive an error.

To create the file system, use the command:

mkfs.s3ql s3://mybucket

It will prompt you for your encryption password. This should be the same as the bucket password in the authinfo file.

Now that your file system is created, you can mount it using the command:

mount.s3ql s3://mybucket /mnt/s3/bucket

Of course, your bucket name and mount point will vary.

Now, if we want to mount this bucket automatically on boot, we need to add an upstart script to /etc/init. Fortunately, s3ql comes packaged with one, s3ql.conf.

I added "--allow-other" to the mount.s3ql command to allow users other than root to access the mounted bucket.

#
# This file can be placed in /etc/init. It defines an upstart job that
# takes care of mounting and unmounting an S3QL file system.
# 
description "S3QL Backup File System"
author      "Nikolaus Rath <Nikolaus@rath.org>"

start on (filesystem and net-device-up IFACE=eth0)
stop on runlevel [016]

env BUCKET="s3://mybucket"
env MOUNTPOINT="/mnt/s3/bucket"

expect stop

script
    # Redirect stdout and stderr into the system log
    DIR=$(mktemp -d)
    mkfifo "$DIR/LOG_FIFO"
    logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" &
    exec > "$DIR/LOG_FIFO"
    exec 2>&1
    rm -rf "$DIR"

    # Check and mount file system
    fsck.s3ql --batch "$BUCKET"
    exec mount.s3ql --upstart --allow-other "$BUCKET" "$MOUNTPOINT"
end script

pre-stop script
    umount.s3ql "$MOUNTPOINT"
end script

After adding this script, in theory you should be able to reboot and have your bucket automatically mounted, but this is were I ran into trouble. Mine wasn't being mounted.

My problem was caused by the fact that upstart was running the script as root, but I'd created the file system as another user. Once I copied the .s3ql directory from the home directory of the user I'd been logged-in as to /root, the problem was solved.

I hope this helps someone else out there. Although I haven't been using my mounted S3 bucket for long, I'm impressed with how my initial tests have gone.

Also, this answer was written about a week after the solution was found. I think I've covered everything, but if you find I've missed a step, let me know and I'll add it. You'll also want to read the manual, its really worth reading if you intend to use s3ql.

Answer 1

With some help from Nikratio, I'm finally able to mount my S3 bucket automatically on boot. I've updated my question with the solution.