Y
Y
yamatoko2015-11-15 08:33:21
VPN
yamatoko, 2015-11-15 08:33:21

How to make a web scraper correctly?

On one of the sites, the API has a limitation that cannot be bypassed. The API is only free. Therefore, it was decided to make a scraper.
I know how to make it, but here's how to make it look more like a person and not be banned?
I have time, that is, my scripter can work for several hours where it would normally take several minutes. This is already a plus. What else can be done? Different combinations of vpn, proxy? What about tor? What else? Maybe there are other ways?
In general, the question is: how to make a person out of a scripter?

Answer the question

In order to leave comments, you need to log in

3 answer(s)
S
sivabur, 2015-11-15
@sivabur

Need more specifics. Access to api by login password or not? What are the restrictions on api? (max. number of requests from one api user) by ip to manage with socks proxies, by account with different accounts.
If you are interested in the maximum per user, then this is that your web scraper executed js + made requests after a random amount of time + emulation of moving the mouse. But do you need a maximum cut for a person. Since you definitely sacrifice speed. And it costs nothing to create 1000 email boxes right now. (If there is an account binding to the mail). Also, if there is ajax, it makes sense not to load the page completely, but send only ajax request. You can also just do not wait for the page code to load until the end. Do not upload scripts and pictures.
Well, usually everything depends on the limitations, they must be studied and competently not exceeded, while looking for workarounds.

T
throughtheether, 2015-11-15
@throughtheether

I know how to make it, but here's how to make it look more like a person and not be banned?
Simple and slow - use Selenium with a browser, pause between requests.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question