Queue system for parsing data with API?

Question

Programming

Good afternoon,
I need to parse user data from the API according to the plan and according to his request. That is, when he logged into the site or clicked the refresh button.
Cron data should, for example, be polled once an hour, and as soon as the user enters the site and does something there, set priority to the beginning.
You also need to limit API requests to no more than 200 requests per second.
How and why can this be done?
I read on RestMQ, but I don’t know if this is what I need or not.
Thanks,
I'll be glad to hear your answers.

Answer 1

Judging by the restrictions, we are talking about the VK API?
on the first question: why not just make a separate request to the API and update the data for a specific user as soon as the user logs in to the site?
on the second question: if you are updating data for a large number of users at once, then in some APIs (VKontakte, for example https://new.vk.com/dev/execute) there are stored procedures. With their help, you will be able to receive updated data for 100 users at once in one request, this will greatly save traffic and reduce the chances of hitting the limit.
Describe in more detail what you want to do, maybe there is a more elegant solution

Answer 2

Alternatively, use RabbitMQ to create a single API request queue. In workers, limit the number of requests per second. The language is not specified, but on php one can use https://github.com/bandwidth-throttle/token-bucket for this.
You can have 2 queues - one for tasks from cron, the second only for priority updates on demand.