What's the best way to store lots of small files?

Question

linux

Good day!
About ~100k zip archives per day will be created daily by standard NodeJS tools.
The size of each archive is <1kb.
Each of them needs to be stored for about a month.
Questions:
1. Is SSD or HDD better to use for this?
2. What is the best file system?
3. How quickly will the disk die from this?
Each archive must be available for download via a link and with a specific name, so the hash method is not very suitable (./e3/b0/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855).

Answer 1

Is SSD or HDD better for this?

HDDs usually last longer and are cheaper per unit of volume, SSDs are faster. So "better" depends on the context.

2. What is the best file system?

You can try Btrfs , there is no limit on the number of files. I can’t say anything cardinally good or bad about this FS yet ... I put it on one combat vehicle for testing, it costs about a year or more, the flight is normal.

3. How quickly will the disk die from this?

In my regular server (small uniprocessor machine with 4 cores) for 5 years, the HDD stood idle, under a decent load, an ordinary WD. Then "out of harm's way" I changed it.

Answer 2

the load on the file system will be very large, even if you use XFS or ZFS. (I would personally recommend XFS). But this is not an engineering path. Store like normal people in the database.

Answer 3

Questions:
1. Is SSD or HDD better to use for this?
2. What is the best file system?
3. How quickly will the disk die from this?

It's better to use a hybrid in your cases.
Use SSD as a cache for HDD + use part of the memory for cache for SSD.
ZFS can do all this.
Again ZFS or XFS. With XFS, a UPS is required, although with FS everything got better, but still.
If you do a bunch, then a normal SSD with a normal garbage collector will die for a very long time.
There is one but

About ~100k zip archives per day will be created daily by standard NodeJS tools.
The size of each archive is <1kb.
Each of them needs to be stored for about a month.

With this approach, I personally would use everything in the database.