I
I
Ivan Hrushka2018-06-19 11:33:56
Python
Ivan Hrushka, 2018-06-19 11:33:56

How to optimize query in postgres with aiopg?

I have such a piece of code that takes 10M records from the database. With each entry, you need to perform a certain action and then write it all back to the database. As you can see from the code, I create a list and throw it all into an event loop. The problem is that after 10 minutes of work the memory runs out and the computer freezes. Maybe someone already faced such a problem.

async def run(self):
        tasks = []
        async with aiopg.create_pool(DNS) as pool:
            async with pool.acquire() as conn:
                async with conn.cursor() as cur:
                    await cur.execute("SELECT id, pin, county from land where pin is not null")
                    async for row in cur:
                        tasks.append(self.loop.create_task(self.task(row, pool)))
        await asyncio.wait(tasks)

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question