Amazon AWS S3 file naming strategy for performance

Question

I am currently utilizing the following naming scheme:

/#{bucket_name}/#{customer_name}/fi/le/na/filename.jpg

So an image for a file named dsca007.jpg for customer bent would be stored here:

/images/bent/ds/ca/00/dsca007.jpg

But I can understand that S3 wants a different form like this:

Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html

Source: https://www.youtube.com/watch?v=uXHw0Xae2ww#t=535

I understand that they want me to change my naming scheme to something like:

/#{bucket_name}/fi/le/na/#{customer_name})filename.jpg

But then he shows this slide in the youtube video:

Does that mean that my first approach was fine?

I really liked the first structure better because I was able to get a list of files owned by one customer, is there a way to list all files under a customer with the last mentioned naming scheme?

It was also quite easy to figure out how much storage each customer was using by running this command:

s3cmd du s3://images/#{customer_name}

How can I do that with the new naming scheme?

Answer 1

If you won't use some unique string when building a key, you'll reach very soon the problem of a key override.

dsca007.jpg is not very unique. there is a big chance that a file with the same name will be uploaded to your s3 bucket. In this case you'll lost the old file or get an error, depends on your configuration.

e.g. this is what we are using: https://s3.amazonaws.com/bucket_name/user_media/videos/screenshots/cmXRyLRQxe9R139023426817_vid001.jpeg

where the prefix: cmXRyLRQxe9R139023426817_ is a self generated string we build and concat to the original file name: vid001.jpg before uploading to s3.