Answer the question
In order to leave comments, you need to log in
How to limit site indexing by Yandex?
Hello. Yandex, like crazy, every day for a week completely indexes all the pages of my site. It's kind of good, but I have an API installed that works every time, and the number of requests for it is limited by the tariff. Before, everything was fine, I indexed a couple of hundred a day and that's it, but now it's 7,000.
How can I limit indexing? I read that you can disable indexing from the main bot, allowing only additional ones and vice versa, will this help?
Answer the question
In order to leave comments, you need to log in
robots.txt
User-agent: Yandex
Crawl-delay: 2.0 # задает таймаут в 2 секунды
User-agent: *
Disallow: /search
Crawl-delay: 4.5 # задает таймаут в 4.5 секунды
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question