How to store a very large number of files? How to store paths in database?

Question

Database

I will have a lot of pictures in my project (user avatars, movie posters, album covers, etc.).
Question: what is the best way to save pictures on the server and how to save paths in the database?
As far as I know, a limited number of files can be stored in one folder, otherwise there may be problems (up to a complete loss of data). Therefore, you need to create some kind of folder tree and scatter files over them.
At the moment, I have not come up with anything better than how to generate a GUID file when uploading and, for example, for 6F9619FF-8B86-D011-B42D-00CF4FC964FF, store photos in the 6F/6F96/6F9619FF-8B86-D011-B42D-00CF4FC964FF.jpg folder. In the database, I plan to store only the GUID, and I will limit the downloadable format for pictures to one JPG. Is it strong crutch?
What other options can you think of?

Answer 1

As far as I know, a limited number of files can be stored in one folder, otherwise there may be problems (up to complete data loss)

no
How to store in the DBMS - whatever you like, you can store the path relative to some directory

Answer 2

That's why GUID?
An auto-incrementing int + salt is enough (so as not to download by id).
And you can store it in storage (Google cloud storage, S3, etc.), they won’t take up space on your server, and backups are made there. And you only have caching within 1-10Gb (depending on the task)

Answer 3

Of course with subdirectories. Path generation method - any.
But I would advise you to make the date the folder names. Like 2017/7/1/hash.jpg - and store this entire path in the database.
When something breaks or you just want to understand what you store there, it will help you a lot. Well, incremental backup is very convenient.

Answer 4

Well, if you need to store a very large number of files directly, then there is a DFS list .

Answer 5

At the moment, I have not come up with anything better than how to generate a GUID file when uploading and, for example, for 6F9619FF-8B86-D011-B42D-00CF4FC964FF, store photos in the 6F/6F96/6F9619FF-8B86-D011-B42D-00CF4FC964FF.jpg folder. In the database, I plan to store only the GUID, and I will limit the downloadable format for pictures to one JPG. Is it strong crutch?

Actually the approach is not new. The same WSUS stores content in this way, although it only needs one level of nesting.

Answer 6

they do it, except that only
6F / 96 / 6F9
sometimes you can throw symlinks into one folder on real ones, arranged in directories

Answer 7

"a lot of pictures" - how much?
tens of thousands - easily live in one folder.
If files are accessed via a direct path, there will be no delay in reading. Recording delay is minimal.
If there are more than 100k files, then in principle yes, you can go as you indicated - create subfolders, but it is desirable to correct the file generation algorithm so that there are not too many folders themselves, because if you have 100k folders of 1-3 files in each - it is inefficient and uneconomical.
In Linux, each folder is at least 3 directory entries.