U
U
Us592019-06-13 12:09:22
Database design
Us59, 2019-06-13 12:09:22

How to control the limit?

There are 8 sites that have request limits, for example site 1 - has a limit of 10 requests, site 2 - has a limit of 30 requests, etc., they are all different.
What are the limits? The fact that from one IP address you can make N number of requests within 24 hours. Haven't figured out how the limits are updated yet (at what time, possibly different times)
How to control the limits? I have 2000 proxies and I need to keep track of how many requests I made from a certain IP address to a certain site. Initially, I noted the limit in the database column. but now it turns out that if I have 8 sites, and each has its own limits, then I need to create 8 columns in the database for each case. Are there easier options to control such things?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
Sergey Gornostaev, 2019-06-13
@Us59

Create table request_logwith fields
and just add entries to it when a request is made by the parser. In such a scheme, it does not matter how many sites and proxies you have.

K
Konstantin Tsvetkov, 2019-06-13
@tsklab

How to control the limit?
You don't need to control the limit. You need the next request to the site to come from a proxy with a minimum number of them for the current day. Using the answer from Sergey Gornostaev this can be done. If everyone COUNTexceeds the limit, postpone the site survey to the next day.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question