How to implement the collection of large amounts of data?

Question

JavaScript

How to correctly implement the architecture for collecting a large amount of statistics?
It will be necessary to perform about 50,000 requests to a third-party API every 15 minutes, take data and write it to the database.
Will a node on the same server be able to handle this at all? What kind of iron is needed for such a volume? How to implement the logic of the siatistics collector itself (as I understand it, setting the setInterval to 15 minutes and looping through the data and making a request is a bad idea)? Will mysql cope with such a load without scaling?

Answer 1

50000/(15*60) ~ 56 requests per second to 3rd party API!
Do men know?

Answer 2

To implement the logic so that you don’t have to throw 50k requests every 15 minutes, and in fact make queues and add tasks to collect information, since this is an external service and its performance is unknown to us, then the queues will help. Yes, and there is a possibility that the collection did not work out and then we start a new one again by the timer, oh, and trash can start here if you do not take into account the confl. requests. Here is a tool for queuing (redis) https://github.com/Automattic/kue