A
A
Alexander Lebedev2018-02-03 10:05:30
Python
Alexander Lebedev, 2018-02-03 10:05:30

How to work with a selection from Mongodb in order to avoid unnecessary calls to the database?

Good morning.
Situation: I get a number of records from Mongodb according to the parameters I need, for example:

tasks_query = task_base.find({
    "$and": [{
        "order_id": x
    }, {
        "site": y
    }, {
        "category": z
    }]
})

Then I need to cycle through this data several thousand times in a loop, making a selection from the Cursor according to a certain parameter, figuratively:
for task in task_query:
    if task['order_id'] = 234:
        return task['category']

The bottom line: I do not want to make several thousand calls to the database, but I want to get one large sample into memory, and then iterate over it. But:
  • If I turn the selection from the base into a list of dictionaries, I will have to check the entire selection for loops each time to find the right one, which is very time consuming and resource intensive
  • When converted into a dictionary, it will be + - the same, because the indices in different samples are different.

Question: if I just save the resulting Mongo Cursor to a variable, can I select from it through find () as from a regular database, without having to access the database every time?
If there is some other solution (for example, another data structure that supports indexes) or I missed something important, I will be grateful for any advice :)

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dmitry Tallmange, 2018-02-03
@sortarage

Mongo is incredibly fast to read. Let it be she who is engaged in searching and sorting, and not python; that's what it's designed for. Don't be afraid to load it with requests.
Finally, make a very simple test. Take the time and describe the required functionality in both versions: monga and python. Measure running time. And everything will fall into place.

L
lega, 2018-02-03
@lega

Then I need to go through the cycles several thousand times ...
It makes no sense, because the result will be only one order_id.
When converted to a dictionary, it will be + - the same
One dictionary - one index, if you need several indexes - make several dictionaries. This is the fastest - O(1) if everything fits into memory. You don't need a special person for this.
But I don't think that it's critical for you in terms of speed, so it's better not to worry and use mongo as it is, it's fast enough.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question