How to solve the problem when parsing robots.txt on webmaster.yandex.ru/robots.xml?

S

Shohruh Uz2014-08-27 13:44:35

robots.txt

Shohruh Uz, 2014-08-27 13:44:35

On webmaster.yandex.ru/robots.xml I will check the sites reg.ru, my.gov.uz without https://. Yandex shows an error: The robots.txt file will not be taken into account by the Yandex robot, because the request is redirected.
But when checking the sites facebook.com, twitter.com does not show an error.
So facebook.com and twitter.com have something set up on the server to handle requests for both http:// and https://. But when logging into the sites facebook.com and twitter.com, access is only available via the https:// protocol.
What is the secret?

Reply

Answer the question

In order to leave comments, you need to log in

1 answer(s)

E

Elena, 2014-08-27
@Nidora

The secret is that there are no redirects due to the fact that the sites work on https only.