AWS EC2 S3 and thousands of directories with a few files in each s3cmd

Question

I have tens of thousands of directories with a few files in each one on a non-AWS VPS (approx 1TB of data). I want to move them all to S3.

I can either zip these in to chunks of 7GB, move (wget or whatever) to EC2 (8GB Ubuntu), unzip and s3cmd them to S3.

OR

go straight to S3 from my VPS with s3cmd sync directory?

which method would be best for performance and reliability?

Thanks

Answer 1

There are two factors that would make the decision for me.

1. What is the avg file size? (Thousands of little files can take more time than several large files)
2. What kind of compression can you get?

If you decide to use an intermediary instance, you can attach a 1 TB EBS volume to handle the files while transferring. It will add a bit of cost, but you wont need to keep the volume once you are done.