Best practice for storing files [closed]

Question

Closed. This question is opinion-based. It is not currently accepting answers.

---

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed 8 years ago.

Improve this question

I am building an application which need to handle stored files uploaded by its users (mostly images and pdf files). sort of an archive app. in order to store the files in most efficent way, i am looking some best practices guidelines in terms of storing files on server.

I've tried to research this subject in google, but i had no luck in finding useful knowledge.
i am quite sure this is very popular topic and i'm just not looking the the right places.

Perhaps somebody can refer me to articles on this matter or provide me some keywords to seek for, in order to answer these questions:

1. Should files be stored and share the same server that used by the application? or should they be stored in a different, dedicated server, separated from the web app's server?
2. what actions need to be done when storing files on server, in order to make it consume as little as possible resources.
   1. Security issues
   2. what are the most popular platforms in this field?

Thanks for your help

Answer 1

It depends upon how many users, how many simultaneous connections, how many files you want. If you aim petabyte data size things are always complex (see e.g. GlusterFS)!

I would generate the file name. You probably want to have hundreds of directories each having a few thousands files (and that depends a lot of the filesystem; ext4 is probably better than ext3). So something like dir034/file12345.jpeg (and store that file path in some database).

^{Some old file systems have a linear complexity for the access of a given file in a large directory. And shells might be unhappy when dealing (e.g. auto-completion) with a directory of million files. Hence my advice to avoid directories with more than several thousands files!}

You might handle very small contents (e.g. less than a kilobyte) differently, perhaps storing them directly in some database row...