How to organize the architecture of a project with 1000 api?

Question

API

Tell me how to organize a project if there is a base and 1000+ endpoint api (each endpoint has several dozen calls) that need to be pulled and filled with this base.
Some APIs rarely need to be pulled once a day, some need to be performed every second.
There are those that are performed for from 30 seconds. and higher. There are those who immediately give the answer.
All APIs are third-party and there is no way to influence them. Please tell me the best way to organize the project.
In my head, I only have to make my own script for each api and put it in cron.
But it seems to me that this is not a very good solution, and there will be connections to the database for each script, and I think it will be difficult to manage all this, but I can’t think of a better way to do it. :(

Answer 1

Use queues. Consumers processing messages from the queue will keep the connection to the database, which is what you are so afraid of. Producers can be called from cron (specified start time), or they can be called from a separate worker (launches at short intervals)

Answer 2

There are ready-made integration systems like tiid or Apache ones. Everything has already been done there. Or you can take a queue broker like Kafka or Rabbit and calmly write a mountain of consumer producers with code reuse ... and a Swager description of customers so as not to die from their endless description ... and for many services, a swager is ready to joke or on the side to generate customers .