How to implement user storage quotas using Amazon S3 as storage like dropbox does

Question

Really need some clever minds to help me with this. I'm stuck! I'm implementing a system where users will be allowed to upload videos and files to my site. I'm currently providing them ftp access to their own ftp user account where they upload the files needed. FTP on linux server supports user quotas.

However, now I need to move my upload protocol away from the ftp protocol and looking into amazon s3 file uploading system via http. ftp just gives too much problems with firewalls, ports, active, passive.. list goes on. :) While http s3 uploading will be just a matter of user allowing internet access to the desktop application that will be doing the uploading.

I've got everything to work fine to the point where I need to set the quota for each user. How do I implement storage quotas on a system like Amazon s3 that does not support it? I can't give unlimited access to my users and I don't want to put my server in the middle either because it just means my server will have to receive the file then upload on the backend to s3 which can be done via mounting s3 as a drive, but then again were back to uploading via ftp to my servers to the mounted s3 drive.

I want my application to upload to s3 server directly but maintain user quotas by uploading to the psuedo amazon folder in 1 bucket. Each user will get a subfolder in a bucket that they upload to.

What would be the implementation / logic to make this work?

Do I put another server to monitor the user by looking up files in a /keyname/ to see how much storage is used by that keyname which would be the useraccount? Wouldn't this cost a lot of money in terms of keeping the status of each account usage up to date?

Help please, I need a strategy. Something similar to dropbox implementation of the storing on s3 yet having quota limits for user uploads.

Answer 1

You will have to keep track of the quota on your own server.

If you have control over the client, you could require them to send the file size to your server prior to the upload to amazon, and deny it if it would exceed the quota.

If you don't, you could allow to upload

Answer 2

If you want User-Based storage AWS WorkDocs might be the solution you are looking for. Creating a Bucket for every user is not an alternative option since the amount of Buckets you can create is very limited.

However, I can't tell you how quotas are implemented.