How to organize regular parsing?

Question

Python

The bottom line is this: the user makes a request to the bot, and he, based on the parsing of 5 sites, gives the result. It takes half a second of parsing to the site and it takes a long time. Parse once every n seconds and give out data that is already stored on the server? What is the best way to do this in python? It seems like urllib3 can stream, but I don’t really know how to organize this. Thanks in advance

Answer 1

1. Parse in parallel (asynchronously) and write the result to the database.
2. To the user - show the preloader until the data is collected.
3. The result of parsing - we always give out from the database.