K
K
Kevin Glare2017-02-10 21:45:56
Python
Kevin Glare, 2017-02-10 21:45:56

How to set cookies or go to the next page in python requests?

Good afternoon, I am writing a parser in python and I ran into a problem:
- there is a site.ru site with subdomains by city
- when you go to spb.site.ru, you are assigned cookies (name: chjs, value: 1, domain: spb.site.ru , path: / etc.)
- if you immediately pass r = requests.get('spb.site.ru/page'), then the site will display data for your city, and not for St. Petersburg.
There are two ways to get the page page - substitute cookies immediately or first go to the page with the city, I tried both methods, but it did not work out. (it seems to me)

def get_page(url):
    r = requests.get(url.rsplit('/',maxsplit=1)[0])
    r = requests.get(url)
    return r.text

and
def get_page(url):
    jar = requests.cookies.RequestsCookieJar()
    jar.set(name='chjs', value='1', domain='site.ru', path='/')
    r = requests.get(url)
    return r.text

While doing:
def get_all_page(html):
    soup = BeautifulSoup(html, 'lxml')
    search = soup.find('title')
    return search

Gives the title of another city

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
danis_2014, 2017-02-10
@KevinGlare

Navigating to another page with requests?

with requests.Session() as session:
    session.post(url_0, data)  # изменяешь свой город
    response = session.get(url_1)  # получаешь данные с другой страницы сайта

1
1011, 2017-02-10
@1011

there is also import cookielib module

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question