P
P
Peter Pitaev2020-10-20 12:48:33
linux
Peter Pitaev, 2020-10-20 12:48:33

Python requests sending requests through proxy at 100% CPU?

Greetings.
I send 10 requests at the same time using python request through a proxy.
While I was sending 8 requests at the same time, everything was fine, my IP was not fired, the CPU load was about 90%.
As soon as I started sending 10 requests, the CPU usage stayed at 100% and my IP burned out.
Is there a possibility that when the CPU is loaded at 100%, some kind of bug occurs and requests are not sent through a proxy?

Example of sending a request:

soup = BeautifulSoup(requests.get(url, proxies=proxies, timeout=10).content, 'lxml')

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
Sergey Gornostaev, 2020-10-20
@sergey-gornostaev

There is an option that the server simply detects too frequent requests.

R
rPman, 2020-10-20
@rPman

I'm not sure that the problem is due to the high load on the processor, by the way, how did you load it that only 8 requests lay down the machine?
An example of how you can burn yourself, you specify an http proxy, and the site redirects you to https using location or exotically to ftp
Does the BeautifulSoup parser accidentally execute xslt requests? some document (url) loads in its own way?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question