Answer the question
In order to leave comments, you need to log in
Is this a bug in robots.txt?
How is robots.txt filled in correctly?
I read earlier that the logic should be like this
. First we write what cannot be scanned by the bot, and then we allow everything that is possible
User-agent: *
Disallow: /cabinet/
Disallow: /cabinet/register/
Disallow: /cabinet/forget/
Disallow: /search/
Allow: /
User-agent: Yandex
Disallow: /cabinet/
Disallow: /cabinet/register/
Disallow: /cabinet/forget/
Disallow: /search/
Allow: /
Host: site.com
Sitemap: https://site.com /sitemap.xml
Answer the question
In order to leave comments, you need to log in
Explicit indication of Allow: / is tantamount to its absence (the default is to allow the indexing of the entire site), so they don’t write :)
The correct robots.txt in your case will be like this:
User-agent: *
Disallow: /cabinet/
Disallow: /search/
Sitemap: https://site.com/sitemap.xml
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question