When parsing a site, ip is blocked due to a large number of requests, how to avoid this?

Question

Python

I wrote a parser for the Avito site, it works based on the bs4 and requests libraries, IP is blocked if requests are too frequent, I tried to make requests through a proxy, it didn’t help, only sleep(random.randint()) helps a little, but sometimes it still blocks, yes and in addition, 100 pages are parsed because of this for quite a long time

Answer 1

Yes, there is no way to avoid it. Either a proxy, or increase the pause between requests .. you can wait. Throw it on the server and let it work for itself

Answer 2

Well, that's right. There is nothing to parse sites. If there are 10 people like you? And if 100? And if 1000? Do you want to buy dozens of new servers for your sake? It's easier to just block the parsers and that's it.