What folder structure should I choose for user-bound files?

Question

linux

The entire folder structure is based on the user id.
those. we take the user id and calculate the folder address from its id.
Thus, the branching structure turned out to be something like this

. No more than 100 folders can be stored in one folder.
It turns out something like the following address of the user's folder:
1/1/1/1/1/
Then we calculate md5 from the id of the file and create an md5 structure in the user's folder. Let 's
say the md5 id of the file is b4k7nbsdgjh45b45mbjdfbgser, then the structure will be like
this
we get the address to the file
1/1/1/1/1/b4k/7nb/sdg/jh45b45mbjdfbgser
I looked at the Habré files are stored in the same way
https://habrastorage.org/files/e4a/69d/525/e4a69d5 .. .
But they do not have users, although they may write in the database which user the file belongs to, but this is not so important.
My portal implies a huge number of files, which over time will require the distribution of files across servers.
I also know that the server has a limit on the number of files and folders.
Have I built the storage structure correctly?

Answer 1

In your case, I would also look towards databases.
Too many folders are also bad - the more nesting, the more overhead will be on the inodes. It all depends on the file system and what you mean by "a lot". From experience, I will say that ext4 copes with a lot of metadata faster than others and is most likely suitable for such a task, XFS, on the contrary, is good for storing very large files - databases, for example.
There is also a new FS - btrfs, it can store metadata for a large file system on SSD drives, and data on slower ones. For large storages, that's it.
Further, you can simply experimentally check what will be faster - a large nesting or many files in one folder. Judging by the structure of the Habr storage, 4096 elements per directory are considered the most optimal.
events.linuxfoundation.org/slides/2010/linuxcon201...
See also What is the maximum number of folders that can be created in a folder?