Scrapy - Python

Question

Python

When parsing one site on Bitrix, after a while, a “400” error is returned from the server. Has anyone dealt with it and can tell you how to more cleverly disguise yourself as a user?
I set the interval between requests to 2 seconds, and the result is always the same - 200 elements and 400 error

Answer 1

It seems to me that this is not because the server is firing that you are walking, you can make a random interval from 2 to 5 seconds to check, maybe the server is just buggy =)

Answer 2

In addition to DOWNLOAD_DELAY and USER_AGENT, which I believe you have already changed, try lowering these CONCURRENT_ITEMS, CONCURRENT_REQUESTS_PER_SPIDER, CONCURRENT_SPIDERS settings.

Answer 3

Maybe there is a limit not on the number of requests per minute or second, but on the number of requests per hour, for example.

Answer 4

The problem was that Bitrix, as always, was ahead of the rest - it stored the history of visits in cookies. And when about 200 elements accumulated, it refused to work as it should. The result - disabled cookies and a few hours of time to analyze the entire problem)