How to think over the architecture of the bot?

Question

Python

Hey!
I need to write a bot for a cart that will parse a certain number of pages and see if there have been any changes. If there were - report it to the group where he is.
The code itself is not the problem. Can't figure out the best way to do this?
the bot works constantly, and the pages need to be checked, for example, once every 10 minutes. sleep does not make sense - the bot will be frozen. Maybe make 2 files? one with a bot, the other with a parser. The parser will be launched by the cron and if it finds changes, it will send information to the bot and it will notify. The idea seems to be not bad, but is it correct and how to implement it between a bot and just a parser script?

Answer 1

threading? Ugh, what am I talking about...

Answer 2

Take advantage of the closure idea, you don't need to waste resources using multithreading.

Answer 3

Maybe make 2 files?

Great idea, but multithreading is not difficult to tie. It all depends on the scale. If this is a mini project, then it's easier to stuff everything into one file. And if a large project, then the bot itself should be decomposed into several files. Also, not every server will give a lot of computing seconds, so the technical side is also worth a closer look. What a tariff not to overpay. Well, etc.
Try to implement any option, and the pitfalls will come out along the way. That's when you write if something goes wrong.

Answer 4

The answer depends on the complexity of parsing, really.
If most of your time in parsing takes up communication with the site, and the cost of parsing itself is minimal, then you can do without multithreading if you use something like aiohttp .
It can work in the same work cycle (reactor) with the rest of the bot, and not suspend it for the duration of the request to the site.
If in parsing a significant proportion of time is occupied by calculations, then multithreading, or even multiprocessing, is already needed here.
Finally, you should use either asyncio.Queue (without threads) or queue.Queue (with threads) to arrange for the parsing results to be passed to the bot. In the first case, the bot can simply await the get() method of the queue in the await loop, and upon receipt of the next block of data, form and send a message.