V
V
Vadim Ivanenko2017-03-14 22:50:31
NoSQL
Vadim Ivanenko, 2017-03-14 22:50:31

How to quickly get a large number of documents by a variety of criteria?

You need to select N documents based on M predicates, where M can be several hundred pieces (up to 1000).
The method input receives an array of filters filters, by which you need to select documents from Monga.

public async Task<IEnumerable<DataDto>> GetDataAsync(IEnumerable<Expression<Func<DataDto, bool>>> predicates)
        {
            var filters = predicates.Select(p => new FilterDefinitionBuilder<DataDto>().Where(p));
            var filter = new FilterDefinitionBuilder<DataDto>().Or(filters);
            var result = await collection.FindAsync(filter) .ConfigureAwait(false);
            return result.ToList();
        }

Each predicate is built as:
a => a.Source.ID == filter.Source
       && a.Dest.ID == filter.Dest
       && a.Type == filter.Type
       && a.Date >= filter.MinDate && a.Date <= filter.MaxDate
       && ...;

The problem is that there are a lot of such predicates and it all works very slowly.
The task is to get the data that is most relevant to the request (there is only one request, but in fact you need to find a suitable document for each predicate). For example, first get documents by ID. If some of them were not found, then we reduce the search accuracy (we use only a few fields and a date range).
Perhaps you need to use a radically different approach (do not use mongu)?
I saw query score in Elastica. I have never worked with it, I know that it is used for full-text search, but the concept with weights for documents seemed to me suitable for my task.

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question