Binary data storage scheme (saving user-uploaded files)

Question

In our app we are currently saving binary data to the database (terrible, I know; but this is legacy stuff and it wasn't my decision). We're trying to migrate this data out to an external device and I'm trying to come up with a scheme to save these files.

We have multiple tenants, I'd like to have a directory per tenant. My scheme for that is to build path using the first-three letters of the tenant. So if you had a tenant called apple, its files would be at /a/p/p/apple. Within these directories, I will save the files. For the files, I want to generate a random 6-character alphanumeric name (only lowercase characters for the time being because internal reasons). So if we generate a file name called 6a8jxo, it will stored as <tenant>/6/a/6/6a8jxo. With this strategy, each tenant can have a maximum of approximately 916 billion unique files (not that we're attempting to save that many), with a maximum of 46,656 files per directory. If I go for a 5-character name, we will have a maximum of 60.5 billion unique files with 1,296 files per directory.

Are there any drawbacks to this approach? I realize that certain directories may only contain one or two files, but in my mind that is not a huge problem.

My colleague doesn't want to do it this way; he wants to use an autoincremented field in the database and then base the directory structure on the hex-value (I assume 32-bit) of that instead of using the tenant. With his strategy the hex value would be used as follows: the file will be stored in the directory located at <tenant>/aa/bbb where aa is the first two characters (most significant nibbles) of the hex value, and bbb is the next three. His reasoning is that he only wants to create new directories when one fills up instead of having numerous, partially-filled directories.

This approach introduces numerous difficulties on the code side of things and I don't see having completely-filled directories as an argument that justifies the extra effort required to do this.

Are there any other strategies or approaches I haven't considered?

Answer 1

I think the main thing you haven't considered is the likelihood of collisions in your random file names.

With such small names, you only have 36 ^ 6 = 2,176,782,336 unique files, which means you are likely to get a collision before hitting 50,000 files (http://en.wikipedia.org/wiki/Birthday_problem) - not a huge number (and of course there's always the chance of getting one much earlier).

I like your colleague's approach simply because it guarantees unique names, regardless of how it affects the filesystem.

If you absolutely don't want to rely on the database to generate a consistent sequence, you can use UUIDs for the names.

You also seem to be planning for very deep trees - how many files (and tenants) do you expect to have? A reasonable rule of thumb is to have 10,000 files (actual, not just possible) per directory, and probably more with modern filesystems. Three levels of sub-directories is almost certainly overkill.

Also, if you do need to split the tenants into multiple directories, hash them first (or use database ids) - using natural names can lead to very unbalanced "buckets" (probably not a big issue here, but it doesn't cost anything to do it correctly).

Finally, how big are the files? Depending on your actual use-case, storing them in the database may be a perfectly reasonable approach.