Binary data storage scheme (saving user-uploaded files)

Question

In our app we are currently saving binary data to the database (terrible, I know; but this is legacy stuff and it wasn't my decision). We're trying to migrate this data out to an external device and I'm trying to come up with a scheme to save these files.

We have multiple tenants, I'd like to have a directory per tenant. My scheme for that is to build path using the first-three letters of the tenant. So if you had a tenant called apple, its files would be at /a/p/p/apple. Within these directories, I will save the files. For the files, I want to generate a random 6-character alphanumeric name (only lowercase characters for the time being because internal reasons). So if we generate a file name called 6a8jxo, it will stored as <tenant>/6/a/6/6a8jxo. With this strategy, each tenant can have a maximum of approximately 916 billion unique files (not that we're attempting to save that many), with a maximum of 46,656 files per directory. If I go for a 5-character name, we will have a maximum of 60.5 billion unique files with 1,296 files per directory.

Are there any drawbacks to this approach? I realize that certain directories may only contain one or two files, but in my mind that is not a huge problem.

My colleague doesn't want to do it this way; he wants to use an autoincremented field in the database and then base the directory structure on the hex-value (I assume 32-bit) of that instead of using the tenant. With his strategy the hex value would be used as follows: the file will be stored in the directory located at <tenant>/aa/bbb where aa is the first two characters (most significant nibbles) of the hex value, and bbb is the next three. His reasoning is that he only wants to create new directories when one fills up instead of having numerous, partially-filled directories.

This approach introduces numerous difficulties on the code side of things and I don't see having completely-filled directories as an argument that justifies the extra effort required to do this.

Are there any other strategies or approaches I haven't considered?

Answer 1

The idea of hashing the directories (not to be confused with generating a hash of the file name) and thus breaking it down into multiple directories is a good one. If its multi-tenant, you probably have a database id for the clients. And its a good idea to track the existence of the files in your db with a unique id (auto-increment or other). So you might simplify your approach and do something like:

/(client id)/(thousands)/(file-id).txt

create a single directory for the tenant, then generate the file path based on the file's unique database id, grouping them in to thousands or tens of thousands. So the file that has an autoincrementing row id of 53445 for tenant number 8 might look like

/8/53/53445.txt

If you need to support a more distributed setup, you can use a uuid rather than an autoincrement field when tracking files. I would not use the name of the client as part of the path, because names change, and you may end up encountering a client with a character in their name that cant be used as part of a file name - just skip all that complexity and use the database id.

Breaking the files down by tenant may not be necessary technically (I usually wouldn't), but may help you if you need to provide those files to the client. Whether its wise to do so really depends on the nature of your application.