How many number_of_shards do you need in an index?

Question

elasticsearch

Hello, how do I determine how many to set number_of_shards when creating an index in elasticseach that will run on a single physical server that will be loaded with 100 million documents with a total weight of several terabytes?
The fact is that this setting cannot be changed after the index has been created and it is not very desirable to re-index later.
Thank you!

Answer 1

For a cluster of one server and a large number of documents - number of shards == number of CPU cores.
If possible, divide the index into several, because. for one server, this is a large base. If you can somehow divide the data - by time, geography, relevance, then this will allow you to search in a specific index, and not in the entire database.

Answer 2

you have to deal with it - Elastic eats memory that that lion
is usually 2-3, 2, in my opinion, it’s written right in the manual, 3 is also somewhere nearby
and then it depends on the number of RPS clients
+ if you chase response time (well, like no more 1ms) for the sake of SEO, a situation arises when gluing from a bunch of shards also takes time, and this causes the number of shards in the cluster, for example, no more than 20
(xs what kind of documents you have, I think in your case the ceiling will be lower)
and that's all , then the distribution of clients goes by clusters