Answer the question
In order to leave comments, you need to log in
How to safely write parsers, grabbers?
Hello guys, how can I safely write a program in which there will be a lot of requests in the form of parsing and grabbing, for 1 - 2 sites?
Let's say I need to parse 100 products and perform 40 - 50 such operations per minute? It's clear that any site will block IP, so you need a lot of proxy addresses that will be updated every 1 - 2 minutes. Do I understand correctly? And where to get so many proxies? Or is it done differently? Thank you for your attention!
Answer the question
In order to leave comments, you need to log in
You understand everything correctly, but there are entire exchanges for proxies and you can buy thousands of them.
PS. If the site is behind a CDN and not one and a half diggers administer it, then proxies will no longer help because machine algorithms work there and monitor a huge number of metrics
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question