Answer the question
In order to leave comments, you need to log in
How to do pagination in Scrapy?
There is such a parser code that collects data but not from all pages somewhere in the middle it stops, tell me what could be the reason
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://www.wildberries.ru/catalog/elektronika/noutbuki-periferiya/kompyutery?sort=popular&page=1',
]
def parse(self, response):
for quote in response.css('div.j-card-item'):
yield {
# 'link': res,
'name': quote.css('span.goods-name::text').get(),
'brend': quote.css('strong.brand-name::text').get(),
'price': quote.css('ins.lower-price::text').get()
}
next_page = response.css('div.pageToInsert a::attr(href)').get()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question