T
T
tispoint2016-03-12 13:56:54
Scrapy
tispoint, 2016-03-12 13:56:54

How to make Scrapy not visit pages with extra parameters in URL?

Site parsing. Donor is a goods store with a developed filtration system.
Scrapy goes into a jungle of links like 2016-03-12 11:48:08
[scrapy] DEBUG: Crawled (200) (referer: site.ru/catalog/server/?brand=123&filter=4187%2C12103%2C12099%2C12141%
2C12109%2C12115%2C12150%2C1207
3%2C12128&onpage=48)
and dies there at infinity.
How to prevent crawling links like &filter=

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
Dimonchik, 2016-03-12
@tispoint

Rule
                 (
                  LinkExtractor(deny=('&filter=',
                                      )),
                  follow = False,
                ),

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question