asyncio task memory limit?

Question

Python

There is a 4 GB data dump, I read the dump line by line and add tasks to the asyncio task. About 2 million tasks are created in memory, which is naturally not enough memory. How to limit asyncio tasks, for example, to 10,000, so that after executing them, new tasks are created from the dump, that is, split reading the dump into parts. The task is to put a huge dump (which must first be edited) into the database.

async def main(path, engine):
    tasks = []
    with jsonlines.open(path) as reader:
        for inx, obj in enumerate(reader, start=1):
            task = asyncio.create_task(load_today_db(engine=engine, obj=obj))
            tasks.append(task)
        await asyncio.gather(*tasks)

Answer 1

The task is quite simple, use queues! https://docs.python.org/3/library/asyncio-queue.html