File system for avatars?

Question

Images

There is a task of storage, hundreds of thousands of small images, I do not want to store in the database. I don't want to go to ext3 either. What do you advise? Maybe there is some kind of daemon or file system that is designed for this?

Answer 1

Google does not give anything, probably there is no ready one. While on the knee was born the following decision.
1) In the main database (mysql) we store all the meta information + the key for the picture. The key is (sqlite file name + id)
2) There are a bunch of small (100-200mb) sqlite files that store images in blobs.
Since the record is append only and the previous ones will not change, only the last sqlite file will get into the daily backup.
What do you say? rave?

Answer 2

And I would honestly advise the base, recently copying my very “small images” - 600k of them there - took a day. And then it was - toli for two, roofing felts for three
gigs. She also worked out tarfs very well - but she was completely ready

Answer 3

a long time ago the answer to this question could be reiserfs

Answer 4

Btrfs + storing several images in one file and indicating the position in the url or somewhere else. Read about the architecture of facebook and contact, there is a lot of useful information.

Answer 5

Consider that avatars are most likely a lot of read requests.
When returning them from the database, you will violate the disk subsystem, while when returning them from the disk, _all_ avatars (depending on the amount of server memory) can be cached by the system in memory and their return will be turbojet.

Answer 6

Pictures need to be stored on a hard drive and no database will do here. creates an unreasonable load on the server. But about how to store it, I would recommend splitting it into subfolders by id-photo in this way: if the photos are < 100k, then /ff/32123e21.jpg is enough. If > 100k && < 2m, then /ff/ff/321312312.jpg, and c /ff/ff/ff/321312312.jpg can go straight into space...

Answer 7

And yet, depending on the needs, you can use a Derby-type database - it itself always consists of dozens of files, regardless of the number of tables in it

Answer 8

presumably this task is for couchDB

Answer 9

Look towards S3+CloudFront. And you don't have to bother with local storage.

Answer 10

MogileFS livejournal was originally created for this purpose.