Optimal number of files per directory vs number of directories for EXT4

Question

I have a program that produces large number of small files (say, 10,000 files). After they are created, another script accesses them and processes one by one.

Questions:

• does it matter, in terms of performance, how the files are organized (all in one directory or in multiple directories)
• if so, then what is the optimal number of directories and files per dir?

I run Debian with ext4 file system

Related

• Maximum number of files/folders on Linux?
• https://serverfault.com/questions/104986/what-is-the-maximum-number-of-files-a-file-system-can-contain

Answer 1

10k files inside a single folder is not a problem on Ext4. It should have the dir_index option enabled by default, which indexes directories content using a btree-like structure to prevent performance issues.

To sum up, unless you create millions of files or use ext2/ext3, you shouldn't have to worry about system or FS performance issues.

That being said, shell tools and commands don't like to be called with a lot of files as parameter ( rm * for example) and may return you an error message saying something like 'too many arguments'. Look at this answer for what happens then.