Generally speaking I create directories like YYYY/MM/DD
to store my files, often with another level for different sources. Sometimes I'll use YYYY-MM/DD
or something similar. Note that there are only 3652 days in a decade, so you could even have a single level like YYYY-MM-DD
and not get directories that are so large that they're hard to work with. If you have a filesystem that indexes directories, you can easily have 10s of thousands of files in a directory, otherwise one thousand should probably be your upper limit.
To process the files, I don't bother doing any actual searching of directory names. Since I know what date I'm interested in, I can simply generate the paths and scan only the directories containing files in the proper date range.
For example, let's say I want to process all files for the past week:
for $date = today() - 7 to today():
$path = strftime("%Y/%m/%d", $date)
for $filename in getFiles($path):
processFile($path, $filename)