Answer the question
In order to leave comments, you need to log in
How to merge query results to Elastic + filter with Mysql?
There is a data set of 300 million rows every day. The data set consists of several fields: one text field, the rest are numeric. There are about 80 million unique values in the text field for all time. How to implement a scheme in which we load these 80 million unique values in elastic, the rest in Mysql. The difficulty is that the search must be done simultaneously both in the text field and in numerical data (the AND condition). For example, we drive a certain query into the elastic, it produces X relevant results, but they do not fit the numerical filter, only the result in position X + 1, for example, fits. If you load all 300 million denormalized data into elastic, the index size becomes very large (several tens of gigabytes, and the data is needed for a month, for example), and they are loaded long enough (they manage to do it in a day, but back to back).
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question