R
R
R4ndolphC4rter2019-07-17 19:47:43
Python
R4ndolphC4rter, 2019-07-17 19:47:43

Python requests how to work with proxy correctly?

I am writing a python program that constantly sends requests to a certain website.

r = s.post(URL, data=data, headers=headers, proxies=proxies)

proxies variable - dictionary
proxies = {
    'http': 'xxx.xx.xxx.xx:xxxx'
}

If I did not change the value in the dictionary and sent requests from one ip, then the server would block me for a while, and I would not be able to receive the correct information from my ip.
Therefore, after each request, I change the value in the dictionary to another one (the list of these values ​​is prepared in advance).
Problem:
Despite the fact that I change the value of proxies, the server still blocks me, and my requests do not go through.
I don't understand what is causing this problem.
I also sent requests to this page icanhazip.com (it returns my ip). Almost always, the displayed result of this page matches the proxy value.
Question:
How can I fix this problem? (I use requests). Or can you recommend any alternative?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
I
Ivan Yakushenko, 2019-07-17
@R4ndolphC4rter

First of all:

proxies = {
    'http': 'xxx.xx.xxx.xx:xxxx',
    'https': 'xxx.xx.xxx.xx:xxxx'
}

Second: are you replacing the User-Agent? By default it is 'python-requests' - many sites block it.
Thirdly: if you take a proxy from the public domain, then it may already be blocked (in fact, this is quite often). Take a large number of proxies at once and try to sort through them like this:
proxies = []
for proxy in proxies:
    response = requests.get(proxies=proxy)
    if response.status_code == requests.codes['ok']:
        break

response.text

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question