Answer the question
In order to leave comments, you need to log in
What are the alternatives to tor/proxy for parsing?
You need to upload ~10.000.000 pages. On the Internet, I collected about 20k more or less live proxies from several sources, enough for a little more than 1,000,000 pages. I connected Tor - after 150,000 pages bans flew, as a result, the script hangs in the loop for more time, recreating the session and trying to break through the ban.
Actually the subject - are there any other ways of parsing, except through a tor / proxy?
Answer the question
In order to leave comments, you need to log in
Spend $20 and buy normal proxies. For example, fine proxy helped me in due time.
10000000/20000=500
Nothing about the number, this page makes 60 requests when loading. You most likely have bans for another reason, and this reason is just lousy proxies. On public proxies, I often notice that they do not allow access to Russian sites, and they also do not allow Russian sites from distant ips.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question