S
S
Soft_touch_plastic2021-06-28 13:35:18
Python
Soft_touch_plastic, 2021-06-28 13:35:18

Why doesn't grequests with proxy work on hosting?

Good afternoon, I'm writing a parser for one site, split it into streams, it works according to the following principle - an array of proxies and an array of links commensurate with it are passed, everything is parsed using grequests. The question is that on my machine (windows 10) everything is parsed normally, here is the code:

proxies = []
    authes = []
    for proxy in raw_proxies:
        host = proxy.split('@')[0]
        auth_arr = proxy.split('@')[1]
        auth = HTTPProxyAuth(auth_arr.split(':')[0], auth_arr.split(':')[1])
        authes.append(auth)
        proxies.append(dict(http='socks5://'+auth_arr+"@"+host, https='socks5://'+auth_arr+"@"+host))
    
    for link, name, proxy, auth in zip(urls, names, proxies, authes):
        rs.append(grequests.get(link, proxies=proxy, auth=auth))

    results = grequests.map(rs)

Everything works fine here, I checked it through 2ip.ru, it returns the ip specified in the proxy. However, when I uploaded this to the hosting, instead of an array of requests 200, it returns an array of None!
Made test code:
def with_proxy():
    proxy = '****:*****@******:*****'
    host = proxy.split('@')[0]
    auth_arr = proxy.split('@')[1]
    auth = HTTPProxyAuth(auth_arr.split(':')[0], auth_arr.split(':')[1])
    proxies=dict(http='socks5://'+auth_arr+"@"+host, https='socks5://'+auth_arr+"@"+host)
    rs = [grequests.get('https://google.com', proxies=proxies, auth=auth)]
    result = grequests.map(rs)
    print('Запрос с прокси->', result)

def without_proxy():
    rs = [grequests.get('https://google.com')]
    result = grequests.map(rs)
    print('Запрос без прокси->', result)

with_proxy()
without_proxy()


Result on my wheelbarrow:
Запрос с прокси-> [<Response [200]>]
Запрос без прокси-> [<Response [200]>]


On two hosts, one is unknown to me, the other is mine, from reg ru:
Запрос с прокси-> [None]
Запрос без прокси-> [<Response [200]>]


What is the reason for this behavior, and is it possible to get around it somehow?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question