W
W
weranda2019-01-26 11:18:10
Parsing
weranda, 2019-01-26 11:18:10

Why is the request not returning a response in Scrapy?

Hello
CODE:

import scrapy
from scrapy.spiders import CrawlSpider
from scrapy import Request


class TestSpider(CrawlSpider):
  name = "test_spyder"
  allowed_domains = ["toster.ru"]
  start_urls = ['https://toster.ru/tag/scrapy/questions']

  def parse(self, response):
    title_1 = response.xpath('//h1/text()').extract_first()
    next_url = 'https://toster.ru/tag/crawling/questions'
    title_2 = Request(url=next_url, callback=self.parse_some)
    yield {'title_1': title_1, 'title_2': title_2}

  def parse_some(self, response):
    return response.xpath('//h1/text()').extract_first()

I don’t understand why a request is returned instead of the second header:
{"title_1": "Scrapy", "title_2": "<Request GET https://toster.ru/tag/crawling/questions>"}

In theory, two headers should be returned: Scrapy and Crawling.
Please indicate the error and how to fix it in a specific code. If possible, do not offer to fill in the meta Request field, as well as the ItemLoader's. I would like to understand what exactly was wrong here.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
P
Pavel Zamyatin, 2019-03-07
@corw

Sorry, but this code is wrong. I wanted to start writing in detail, but this will be a whole article and not an answer.
Try redoing the scrapy tutorial step by step: https://doc.scrapy.org/en/latest/intro/tutorial.html

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question