Amazon S3 - directory structure for storing small statics (eg pictures)

Question

Amazon Web Services

Comrades, tell me pliz: from linux's experience, I made a rule that if a project can have hundreds of thousands or even millions of pictures, then they cannot be kept in one directory. At least not to run out of inodes.

Now, more and more often there are thoughts about moving statics to amazon s3, in connection with which a reasonable, as it seems to me, question-statement arises: should inodes not run out there?!

Actually the question is - what could be wrong if a lot of pictures (starting from 500K and above) are unloaded into one directory inside the basket?

ps By the way, it's better not to put a bunch of files in the root of the basket - then you get tired of deleting the basket (if you suddenly need it, of course).

Answer 1

If you want to move to Amazon just because there are a lot of files and you are afraid of problems with the system because of this, then you can simply use another file system where there are no such problems. For example, XFS: everything depends only on disk space.

Answer 2

inodes are attached to the file system, i.e. can end only for a certain section. How the files will be stored, in the same folder or different ones, does not matter.

Answer 3

Well, for everything I can use simple sharding, I store
the first hundred thousand images in a directory, say 1
second in directory 2
, and so on
in the future it is easy to make all sorts of changes and distribute them to servers.

Answer 4

If you look a little further, the question arises - why store pictures on Amazon?
We just had a project that mixed servers rented in a regular data center and servers in Amazon EC2.
So, if your Amazon nodes work with a more or less even load 24/7, then renting a server in the data center may be more profitable.
Just an ideal use for Amazon nodes is when you run them for a short time for computational tasks (say we run there on 15 nodes the calculation of forecasts in some modeling task that has little time to complete. Or when you connect servers when in peak your own fail.