How to implement the work of the parser on the site?

Question

Django

In general, there is a task to create a site that will display a table with data that is parsed from another site. But parsing all the data from this site takes more than 20 hours (you need to set timesleep so that the site does not block the parser). Therefore, it is necessary to implement it in some way so that the parser works in the background, and at this time fresh data is shown on my site (as it is being parsed). I have not worked with serious projects, so there is no understanding of how to implement it all correctly.

So far, there is only one option: Run two programs, one will have a django project with a website, the second will have a parser that will receive data and write it to the database. And the site will unload from the database and show them at home. But it will have to run two consoles at the same time. In principle, of course, this is not a particular problem, but suddenly there is a way to implement all this in one program.

ps Please do not offer advice like: "why do you need this", "use a proxy", etc.

There is a specific parser and a web application, if there is some way to connect them somehow, then I will be very glad to hear the options. Thank you!

Answer 1

Run two programs, one will have a django project with a website, the second will have a parser that will receive data and write it to the database. And the site will unload from the database and show them at home. But it will have to run two consoles at the same time.

Have you heard of systemd? Well, or at least about the service?
Or does your site work exactly as long as you have an ssh session open to your server?
We need a separate daemon that parses the data and adds it to the database.

Answer 2

What's wrong with using a proxy? I apologize
the same as explaining how to write code, but please do not offer advice to use your hands and brains
. Then do dynamic parsing. You are asked for x/product/15 and you immediately ask for y/product.php?id=15. One dick hit the limit