Answer the question
In order to leave comments, you need to log in
How to optimize query in postgres with aiopg?
I have such a piece of code that takes 10M records from the database. With each entry, you need to perform a certain action and then write it all back to the database. As you can see from the code, I create a list and throw it all into an event loop. The problem is that after 10 minutes of work the memory runs out and the computer freezes. Maybe someone already faced such a problem.
async def run(self):
tasks = []
async with aiopg.create_pool(DNS) as pool:
async with pool.acquire() as conn:
async with conn.cursor() as cur:
await cur.execute("SELECT id, pin, county from land where pin is not null")
async for row in cur:
tasks.append(self.loop.create_task(self.task(row, pool)))
await asyncio.wait(tasks)
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question