P
P
Pavel2020-05-27 20:55:29
Python
Pavel, 2020-05-27 20:55:29

Python requests post pagination, how to navigate to the next page?

All good evening
I am making a parser and the task was to navigate through the pages of the site. When I click on the page number in pagination, the page reloads, but when I navigate through requests.post(url), I get a 404 output.
In Chrome DevTools, it issues two requests.

First

example of my code

rom bs4 import BeautifulSoup
import requests

import config as conf

headers = {
    'user-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0',
    "Accept-Language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7"
}

data = {
    'authenticity_token': '',
    'email': conf.LOGIN,
    'password': conf.PASS
}

session = requests.Session()
session.get('https://www.strava.com', headers=headers)

def get_token():
    response = session.post(conf.URL_AUTHORIZATION)
    soup = BeautifulSoup(response.text, "lxml")
    token = soup.find('input',{'name':'authenticity_token'}).get('value')
    return token

def authorization():
    print(get_token())
    data['authenticity_token'] = get_token()
    session.post(conf.URL_AUTHORIZATION, headers=headers, data=data)

def get_response():
    url = 'https://www.strava.com/segments/9926585/leaderboard?club_id=225082&filter=overall&page={}&per_page={}&partial=true'.format(2, 25)
    response = session.post(url, headers=headers)
    print(response)

# Точка входа
def main():
    authorization()
    get_response()

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dr. Bacon, 2020-05-27
@bacon

Campaign, only ignoramuses and bots remained

G
galaxy, 2020-05-27
@galaxy

Why post and not get?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question