P
P
PRAIT2019-07-01 19:20:29
Parsing
PRAIT, 2019-07-01 19:20:29

How to safely write parsers, grabbers?

Hello guys, how can I safely write a program in which there will be a lot of requests in the form of parsing and grabbing, for 1 - 2 sites?
Let's say I need to parse 100 products and perform 40 - 50 such operations per minute? It's clear that any site will block IP, so you need a lot of proxy addresses that will be updated every 1 - 2 minutes. Do I understand correctly? And where to get so many proxies? Or is it done differently? Thank you for your attention!

Answer the question

In order to leave comments, you need to log in

3 answer(s)
I
Ivan Shumov, 2019-07-01
@inoise

You understand everything correctly, but there are entire exchanges for proxies and you can buy thousands of them.
PS. If the site is behind a CDN and not one and a half diggers administer it, then proxies will no longer help because machine algorithms work there and monitor a huge number of metrics

A
Alexander, 2019-07-01
@NeiroNx

Yes, no one will block anything. It's not even DoS. You first write a program that will be able to make requests at such a speed - you need at least 10 threads per vskidka.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question