B
B
Banki002020-05-16 07:08:46
Python
Banki00, 2020-05-16 07:08:46

Pagination when parsing Avito in python?

The problem with parsing subsequent pages, in the version to which I finished, I can only pull out the first page, if you use this option pagination = soup.find('div', class_='pagination-root-2oCjZ'), then everything seems to be fine , the entire list of pages is displayed in html, but I don’t understand how to pull them out ... The pages themselves - https://www.avito.ru/murmanskaya_oblast/avtomobili...

import requests
from bs4 import BeautifulSoup

def get_pages_count(html):
    soup = BeautifulSoup(html, 'html.parser')
    pagination = soup.find('span', class_='pagination-item-1WyVp').findNext('span')['data-marker']
    if pagination:
       return int(pagination[-2])
    else:
        return 1



def parse():
    html = get_html(URL)
    if html.status_code == 200:
        cars = []
        pages_count = get_pages_count(html.text)
        for page in range(1, pages_count + 1):
            print(f'Парсинг старницы {page} из {pages_count}...')
            html = get_html(URL, params={"p": page})
            cars.extend(get_content(html.text))
        print(cars)
    else:
        print('Error')


parse()

Answer the question

In order to leave comments, you need to log in

2 answer(s)
B
Banki00, 2020-05-17
@Banki00

Generally done this way

def get_pages_count(html):
    soup = BeautifulSoup(html, 'html.parser')
    pagination = soup.find('div', class_='pagination-root-2oCjZ')
    line = pagination.text
    p_count = int(line[-8])
    if p_count > 1:
        return p_count
    else:
        return 1

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question