asyncio. How to add a task to a running loop?

Question

Python

Hello, I have one question. There is a parser that pulls information from several sites. There is a problem in the speed of loading this data, since there are many links that need to be processed. Because of this, I decided to use asyncio and aiohttp. So, the question is how can I make this parser, for example, send data to asyncio as needed, and it will process it. That is, so that in this allowable class, the loop was launched by the run_forever () method and some kind of function that added a task to the loop. Is it feasible? And is the approach right? Thanks in advance!

Answer 1

If I understood everything correctly, then you should have:
The idea is the following. We write an executor coroutine that takes the next task from the task queue for downloading, downloads what is required, and puts the result into another queue of results. Using the loop.create_task call, we create several coroutine executors working in parallel, passing them a queue with tasks and a queue of results as a parameter. We write a task generator coroutine, which, in accordance with your algorithm, will fill the queue with tasks. Something like this, unfortunately now there is no time to write code.
There is such a report on the topic - Patterns for asyncio applications , it describes several options for launching cycles, including in a separate thread as you need. Dare :)

Answer 2

https://github.com/niklak/aiohttp_spider
and look for an article somewhere the author wrote,
but in general - Scrapy would make life much easier