How to properly organize the Worker Pool structure?

Question

go

Hello!

Tell me how to properly organize the application architect to solve the following problem:

In BV, with a frequency of + -200 records per minute, tasks are inserted for "almost parsing" (url, where to send requests and their number).
It is important that one request must be from 1 proxy address and then the proxy must be idle for 0.5 seconds to protect against a ban.
The server supports 500 concurrent requests over the network.
I need an architecture that will allow me to control the rate of requests per minute for each task.

That is, I have 500 threads, there are 10k proxies and 200 new tasks per minute.
If the task is to access the URL 100 times and the speed is 50 per minute, then I can spend 50 proxies in one iteration and then I have to wait a while so as not to increase the speed limit.

It is not yet entirely clear how to correctly control the distribution of proxies among tasks and control the speed of tasks.

Answer 1

Use Rate Limiting algorithms - https://en.wikipedia.org/wiki/Rate_limiting
Rate limiter implementation package: https://godoc.org/golang.org/x/time/rate

Answer 2

It's called throttling.
You can send tasks one by one to the queue, and take them from there using the rate limit mechanism (google).
If there are multiple service instances, then Redis can be used .
More information:
https://docs.microsoft.com/en-us/azure/architectur...