Answer the question
In order to leave comments, you need to log in
Strange robots.txt, what to do with it?
Good day, forum users.
Today, on one site, I saw the following lines in robots.txt:
User-agent: *
Disallow: /
Allow: /$
Allow: /ua/$
Allow: /en/$
As far as I understand " User-agent: * | Disallow : / " prohibits the site from being indexed by both Google and Yandex, but the lines " Allow: /$ | Allow: /ua/$ | Allow: /en/$ " allow indexing the main page in 3 languages. How professional is this solution or is it more like nonsense?
It's just that such lines are incomprehensible to me, if someone can help and explain, I will be grateful.
Answer the question
In order to leave comments, you need to log in
If all useful pages are available to search engines, and all useless ones are not, then this is the correct robots.txt. You can check whether pages are available through the verification tools in Google and Yandex webmasters.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question