Downloading very large number of files from S3

Question

I would like to setup a Disaster Recovery copy for an s3 bucket with ~2 million files.

This does not have to be automated since we trust Amazon's promise for high reliability, we have enabled versioning and setup MFA for deleting the bucket itself.

So I just want to periodically download (manually) the contents of the bucket to keep an offline copy.

I have tried a few S3 clients but most of them hang when dealing with such a large folder.

Is there any tool that is right for the job or do we have to resort to Amazon's data export service (where we have to send them a usb drive everytime we need an offline backup).

thanks in advance for your advice!

Answer 1

Dealing with bucket with millions of files can be very challenging unless there is some sort of 'structure' to your file names. Unfortunately this wont help any of the GUI tools so you're stuck implementing your own solution. For example:

1. If all your files start with a date, you can use the marker header in a Get Bucket request to only return files older than a certain date.

2. If your files are arranged in 'virtual' folders, you can user the prefix and delimiter headers to process each folder separately. (Consider doing this in parallel to speed things up)

Even if you have no structure all is not lost. The S3 clients hang because they are trying to hold the entire 2 million file listing in memory. You could download the object listing 1000 files at a time but save this to a file/database etc. It'll take a long time to get all 2 million but once you are done just loop through your saved list and download as necessary.

Better yet, if you are able to 'index' your files in a database as they are added to S3, you can just use that to determine which files to download.

Answer 2

You can use the jets3t library with Java to build your own tool. If you know core Java it is not hard to use.

http://jets3t.s3.amazonaws.com/toolkit/toolkit.html

There are some code samples.

http://jets3t.s3.amazonaws.com/toolkit/code-samples.html