Answer the question
In order to leave comments, you need to log in
How to control the limit?
There are 8 sites that have request limits, for example site 1 - has a limit of 10 requests, site 2 - has a limit of 30 requests, etc., they are all different.
What are the limits? The fact that from one IP address you can make N number of requests within 24 hours. Haven't figured out how the limits are updated yet (at what time, possibly different times)
How to control the limits? I have 2000 proxies and I need to keep track of how many requests I made from a certain IP address to a certain site. Initially, I noted the limit in the database column. but now it turns out that if I have 8 sites, and each has its own limits, then I need to create 8 columns in the database for each case. Are there easier options to control such things?
Answer the question
In order to leave comments, you need to log in
Create table request_log
with fields
and just add entries to it when a request is made by the parser. In such a scheme, it does not matter how many sites and proxies you have.
How to control the limit?You don't need to control the limit. You need the next request to the site to come from a proxy with a minimum number of them for the current day. Using the answer from Sergey Gornostaev this can be done. If everyone
COUNT
exceeds the limit, postpone the site survey to the next day.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question