Why is Connection aborted when connecting to an existing Yandex Weather URL?

Question

Python

import requests

city_id = 27612
url = 'https://export.yandex.ru/weather-ng/forecasts/%s.xml' % city_id
response = requests.get(url)

When trying to connect to the weather API, very often ('Connection aborted.', BadStatusLine("''")) is issued when testing locally, but always when trying from the server.
What is this, Yandex protection? If so, then why is this API, and if not, what am I doing wrong?

Answer 1

Apparently, YandexPogoda takes your program for a bot and resets the connection. To avoid this, you need to pretend to be a browser, i.e. make sure that the http request sent by the program has a User-agent header. Plus, it may be necessary to request data several times for the same reason (a RemoteDisconnected error pops up). I used sessions, it turned out.

import requests
headers = {'User-agent':'Mozilla/5.0','Referer':'http://www.python.org/'}
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(max_retries=10)
session.mount('http://', adapter)
# запрашиваем нужные урлы
for url in urls:    
    response = session.request('GET', url, headers = headers)
    # делаем что-то с response...
    reponse.close()
session.close()

There may still be problems due to the old version of Python, I wrote in 2.7 via urllib2 - it didn’t work, in 3.5 - everything is fine.