Answer the question
In order to leave comments, you need to log in
Python requests sending requests through proxy at 100% CPU?
Greetings.
I send 10 requests at the same time using python request through a proxy.
While I was sending 8 requests at the same time, everything was fine, my IP was not fired, the CPU load was about 90%.
As soon as I started sending 10 requests, the CPU usage stayed at 100% and my IP burned out.
Is there a possibility that when the CPU is loaded at 100%, some kind of bug occurs and requests are not sent through a proxy?
Example of sending a request:
soup = BeautifulSoup(requests.get(url, proxies=proxies, timeout=10).content, 'lxml')
Answer the question
In order to leave comments, you need to log in
There is an option that the server simply detects too frequent requests.
I'm not sure that the problem is due to the high load on the processor, by the way, how did you load it that only 8 requests lay down the machine?
An example of how you can burn yourself, you specify an http proxy, and the site redirects you to https using location or exotically to ftp
Does the BeautifulSoup parser accidentally execute xslt requests? some document (url) loads in its own way?
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question