V
V
Vlad2017-04-06 20:31:47
Yandex
Vlad, 2017-04-06 20:31:47

Trouble with robots.txt?

Hello! I can’t figure out what’s wrong with the robots.txt file
Yandex swears at it, writes:
A contradictory Host directive is set in the robots.txt file
The Host directive specifies a domain where there are no similar instructions in the robots.txt file. In order for the instructions of the Host directive to be taken into account, identical directives must be present in the robots.txt files of all site mirrors.
Check the robots.txt file and resolve any inconsistencies.
The robots.txt file itself is:
User-Agent: *
Disallow: /login/
Disallow: /registration/
Disallow: /feed/
Disallow: /admin/
Disallow: /stream/
Disallow: /rss
Disallow: /search/$
Disallow: /search /topics/$
Disallow: /search/opensearch/$
Disallow: /my/$
Disallow: /blog/draft/
Disallow: /content/draft/
Disallow: /draft/
Disallow: /comments/
Disallow: /tag/$
Disallow: /talk/
Disallow: /profile/
Disallow: /subscribe /$
Host: https://site.ru
Sitemap: https://site.ru/sitemap.xml

Answer the question

In order to leave comments, you need to log in

3 answer(s)
A
Andrey Burov, 2017-04-06
@BuriK666

Do you have the site.ru domain, and not www.site.ru?

V
Vasya Petrov, 2017-04-06
@VasyaPertrov

The Host directive specifies a domain where there are no similar instructions in the robots.txt file

I translate from Russian into Russian: the robots slipped into the bot differ from the fact that you get the bot from the domain specified in the host.

V
Vlad, 2017-04-06
@vlad2189

That's the problem, that the one that is specified is the main domain!
I have a redirect from http to https + from www to without www = https://site.ru
Also in Java I indicated that the site without www

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question