A
A
Andrew2022-01-07 04:00:45
Scrapy
Andrew, 2022-01-07 04:00:45

How to disable redirect in Scrapy?

I'm trying to crawl a site in the en domain, but when parsing, the request is redirected to the ru zone, and the pages of the site in Russian are collected accordingly. Tried to put

custom_settings = {
        'REDIRECT_ENABLED': False
    }

but it did not help, what else can I see / check?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
al_gon, 2022-01-07
@al_gon

https://doc.scrapy.org/en/latest/topics/downloader...

meta={'dont_redirect': True,"handle_httpstatus_list": [302]}

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question