I
I
iva36822020-07-24 22:28:26
MySQL
iva3682, 2020-07-24 22:28:26

How to merge query results to Elastic + filter with Mysql?

There is a data set of 300 million rows every day. The data set consists of several fields: one text field, the rest are numeric. There are about 80 million unique values ​​in the text field for all time. How to implement a scheme in which we load these 80 million unique values ​​​​in elastic, the rest in Mysql. The difficulty is that the search must be done simultaneously both in the text field and in numerical data (the AND condition). For example, we drive a certain query into the elastic, it produces X relevant results, but they do not fit the numerical filter, only the result in position X + 1, for example, fits. If you load all 300 million denormalized data into elastic, the index size becomes very large (several tens of gigabytes, and the data is needed for a month, for example), and they are loaded long enough (they manage to do it in a day, but back to back).

Answer the question

In order to leave comments, you need to log in

1 answer(s)
E
Eugene, 2020-07-25
@Nc_Soft

Load into elastic only what you index by, then any query from elastic returns ONLY record ids, and by id you get full records from mysql by query select * from tbl where id in (1,2,3)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question