Setting up nginx to serve popular files from SSD

Question

Nginx

Friends, I'm puzzling over a somewhat non-trivial task.
Bottom line: there is a file hosting service. Storage servers are equipped with SSD drives. There is a plan to put frequently requested files on the SSD and give them away from them. The calculation of the popularity of files, copying and deleting those that are no longer popular (SSD sizes, alas, are limited) will be handled by the script automatically.

Problem - if the script worked and deleted the file at the moment when someone downloads it, the connection will be terminated with an error.

Suggest how to implement the mechanism of temporary locks of active files for deletion, or maybe there is some magic way to force nginx to look for a file in another location (from the HDD) when it disappears from the current location? The latter, of course, sounds more like a fairy tale, but you never know ...

Answer 1

>Problem - if the script worked and deleted the file at the moment when someone downloads it, the connection will fail with an error.
It won't break because in Linux the file will be accessed via an open file descriptor. It is easy to check if you put the file to listen and then delete it (we are talking about the ext system), so the player will calmly play the track further, and only when it is accessed again will it give an error that the file was not found.

Answer 2

Not quite a beautiful option, but it will be fast and autonomous.
The bottom line is that you need two nginx.
nginx1 makes proxy_cache and adds files to ssd as needed
nginx2 gives files to the first one from hdd
After filling the cache, you will always have the most relevant files in the cache, less relevant ones will fly out of it from time to time.
For your option, you can do it with try files and rewrite to the location that returns from hdd

Answer 3

For questions about configuring nginx, I recommend contacting the official mailing list directly . There is the maximum concentration of people who are very competent in this matter, incl. and the author himself often answers. Get the most correct option.

Answer 4

Are you sure the connection will fail with an error? If Linux, then open files (for example, video) after deletion continue to work (play, until the file is closed, apparently), as far as I remember. So maybe it's just not a problem.

Answer 5

nginc.conf
Enable cache
proxy_cache_path /mnt/tmp/cash levels=1:2 keys_zone=cache:1m max_size=200M;
proxy_temp_path /mnt/tmp 1 2;
accordingly, you prescribe in the location what and how to cache