How to organize the storage of photos for the site?

Question

Node.js

It is supposed to store files on its own server (NodeJS).
The volume is several tens of thousands, that is, the simplest option to dump everything in one directory is no longer suitable. In addition, for image files, you need to store thumbnails.

I now see something like this algorithm after the file is received from the client:
1) Calculate the hash from the image
2) Determine the path to the folder from the hash (as described here , the second method)
3) Make the necessary tubes and save it all side by side. More or less like this:

../6a/c1/e68d51251ab2d826ce8d0b92dd08a7e01c3d.jpeg
../6a/c1/e68d51251ab2d826ce8d0b92dd08a7e01c3d_100x100.jpeg
../6a/c1/e68d51251ab2d826ce8d0b92dd08a7e01c3d_200x200.jpeg

The first question is general: how relevant is this method and does it have worthy alternatives?

And second: how to implement the file structure itself?

fs.writeFile('/home/nodejs/project/static/6a/c1/e68d51251ab2d826ce8d0b92dd08a7e01c3d_100x100.jpeg, buffer, err => {
            if (err) console.error(err.message);
        });

The code shown above will, of course, throw an error, because the directory ../6a/c1/ does not yet exist. Solve by a banal check for the presence of a directory? Or maybe create directories in a different way?

Answer 1

Upload files not to a folder, but to an s3-compatible storage (so that if you wish, you can switch to Amazon or some Linode without suffering and changes in the code) - MinIO . Those. generate some id, turn it into a hash, generate a key and pour it into MinIO. And keep only the originals.
As for miniatures, they do not need to be cut and stored. Imagine if the dimensions in the design have changed - what then? And if a webp-version was required for each thumbnail, should all images be processed again? There is a solution - imgproxy. This is a special proxy service (you can install it on your server, just like MinIO), which cuts pictures "on the fly" at the moment of access (by the parameters in the url). In addition to cutting, there are a lot of different possibilities. There is no internal cache, i.e. before imgproxy, it makes sense to put Nginx for caching thumbnails (so that it doesn’t cut each time).

Answer 2

There are a lot of options, each of which depends on a bunch of nuances and subtleties of the project.
There is only one point - the storage directory should not contain a large amount of files, which is achieved by distributing files between subdirectories.
The principle of creating subdirectories is not important, you can create by dates, for example, such as /16112021/ or more branchy 2021/11/16/, as well as thumbs - you can create it as a hash + size, or you can create everything as the same name (the same hash) , but stored in subfolders of size, such as: /16112021/100x100/e68d51251ab2d826ce8d0b92dd08a7e01c3d.jpeg, which will simply allow you to work with thumbs, knowing only the full-size picture.
In short, there are a lot of options, it’s hard to say which one suits you, just follow logic and common sense.

Answer 3

Here is one of the implementations
https://qna.habr.com/q/673339
You can also try azure blob storage.

Answer 4

I also think the second way is the best. Once I looked at different options and settled on this. A unique ID is generated, directories are created for even distribution according to the second method from stackoverflow... and then you can look at the links (w - width, height depends on the image proportions):

// ID: 3d02573d-4d28-7144-9f6e-f7372a084069
// Путь: [uploads]/[первые два символа ID]/[след. два символа ID]/[ID]/[что угодно][разделитель][разрешение][расширение]
{
  ...
  photos: [{
    original: '/uploads/3d/02/3d02573d-4d28-7144-9f6e-f7372a084069/3d02573d_4032x3024.jpg',
    w1024: '/uploads/3d/02/3d02573d-4d28-7144-9f6e-f7372a084069/3d02573d_1024x768.jpg',
    w600: '/uploads/3d/02/3d02573d-4d28-7144-9f6e-f7372a084069/3d02573d_600x450.jpg',
    w256: '/uploads/3d/02/3d02573d-4d28-7144-9f6e-f7372a084069/3d02573d_256x192.jpg',
    w56: '/uploads/3d/02/3d02573d-4d28-7144-9f6e-f7372a084069/3d02573d_56x42.jpg'
  }]
  ...
}

You can delete the photo along with the thumbs at a time by deleting the folder 3d02573d-4d28-7144-9f6e-f7372a084069 .