How to take a large amount of data over a period of time?

Question

Flask

I wanted to write a simple website with statistics, I put it together, everything was ok.
But how to use python to take data 24 hours a day, if there are a large number of them? Maybe what materials to read or ready-made examples of implementation are there?
I use flask to display data using an array in js.

I take data from an Instagram post and need to build a graph on my own website by the number of comment likes in the timeline.

Answer 1

In general, Flask has nothing to do with it.
You need to figure out the architecture of this kind of task
. What you have already done is just a data view (View) layer.
This is all cool, but this part should not be involved in collecting data. She must read them from some repository in a ready-made form.
Data collection should be handled by another system.
If you want a python and don't really want to invent bicycles, I would recommend taking scrapy and building the necessary scrapers there. It is also possible to organize a pipeline in it - in which to determine what to do with the received data.
There are many ways, but I would recommend you the simplest - write data from scrapy to a database such as mysql, and from there read the data in your flask application.
Depending on the volumes, it may turn out that the proposed solution is not effective.
Since in fact you are interested in collecting metrics, in combat conditions I would recommend using specialized tools for collecting and displaying metrics (for example, the graphite-grafana bundle)
It is unlikely that your mysql-self-written webmord bundle will surpass them in terms of performance, quality and stability.
So everything really depends on the volume and category of the task (I mean "it will go into production tomorrow" or "I'm writing for myself")