Answer the question
In order to leave comments, you need to log in
How to make Scrapy not visit pages with extra parameters in URL?
Site parsing. Donor is a goods store with a developed filtration system.
Scrapy goes into a jungle of links like 2016-03-12
11:48:08
[scrapy] DEBUG: Crawled (200) (referer: site.ru/catalog/server/?brand=123&filter=4187%2C12103%2C12099%2C12141%
2C12109%2C12115%2C12150%2C1207
3%2C12128&onpage=48)
and dies there at infinity.
How to prevent crawling links like &filter=
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question