A
A
Alexey2021-12-25 09:15:55
Python
Alexey, 2021-12-25 09:15:55

How to do pagination in Scrapy?

There is such a parser code that collects data but not from all pages somewhere in the middle it stops, tell me what could be the reason

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://www.wildberries.ru/catalog/elektronika/noutbuki-periferiya/kompyutery?sort=popular&page=1',
    ]



    def parse(self, response):
        for quote in response.css('div.j-card-item'):

            yield {
                # 'link': res,
                'name': quote.css('span.goods-name::text').get(),
                'brend': quote.css('strong.brand-name::text').get(),
                'price': quote.css('ins.lower-price::text').get()
            }

        next_page = response.css('div.pageToInsert a::attr(href)').get()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question