A
A
Artem2014-11-21 12:08:43
MongoDB
Artem, 2014-11-21 12:08:43

How to organize a quick search through 3 billion documents?

Hello
Life pushed me with the need for a quick search among 3 billion documents. Roughly speaking, this is a list of key queries from PS users, from which you need to select those that include, for example, the word "download".
I uploaded a demo billion into MongoDb (by the way, I received 239 GB of data in 139 files) and tried to make a selection through Matches (I work with Mongo through the c # driver).
Even the simple Count of this query took so long that I didn't have the patience. At the same time, 10% of the processor and a bit of memory were occupied by the mongo.
And I would like all this to be done in seconds, not minutes or hours.
Tell me, in which direction to dig?
Thank you.

Answer the question

In order to leave comments, you need to log in

3 answer(s)
E
Eugene, 2014-11-21
@devspec

elasticsearch

B
brutal_lobster, 2014-11-21
@brutal_lobster

Towards mongo index or text search engines - lucene, sphinx..

L
lega, 2014-11-21
@lega

Roughly speaking, this is a list of key queries from PS users, from which you need to select those that include, for example, the word "download".
If the list of keywords is not large, then you can make an indexed array in the documents and put these keywords (or their identifiers) there.
Otherwise use sphinx/elasticsearch. You can also use text index from mongoDB, but it seemed to me too gluttonous.
I tried to make a selection through Matches
At the same time, it enumerates and checks all documents, which is why it takes so long.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question