Backing up to S3: symlinks

Question

I'm trying to figure out right now how to backup some data to S3.

We have a local backup system implemented using rsnapshot and that works perfectly. We're trying to use s3cmd with the --sync option to mimic rsync to transfer the files.

The problem we're having is that symlinks aren't created as symlinks, it seems to be resolving them to the physical file and uploading that instead. Does anyone have any suggestions as to why this would happen?

I'm wondering if I'm missing something obvious or that S3 just isn't suited to this sort of operation. I could setup an EC2 instance and attach some EBS but it'd be preferable to use S3.

Answer 1

There is no symlinks available on S3 but what you can use is google's solution which creates a based file system using S3 (FUSE) . More information here :

https://code.google.com/p/s3fs/wiki/FuseOverAmazon

and here:

http://tjstein.com/articles/mounting-s3-buckets-using-fuse/

I hope it helps

Answer 2

Amazon S3 itself doesn't have the concept of symlinks, which is why I suspect s3cmd uploads the physical file. Its a limitation of S3, not s3cmd.

I'm assuming that you need the symlink itself copied though? If that's the case, can you gzip/tar your directory with symlink and upload that?

Answer 3

Try using the -F, --follow-symlinks OPTiON when using sync. This worked for me.