A
A
Andrew2015-09-17 12:25:02
Search Engine Optimization
Andrew, 2015-09-17 12:25:02

How to check for errors in the Robot.txt file?

Please see the screenshot. Do you need to specify all search engines, or whatever it is called correctly, in User-agent: * ? Yandex, Googlebot
Haven't updated the robot.txt file yet, I'm afraid to spoil it. There is nothing superfluous or maybe I forgot something? I will be very grateful. Sitemap need to specify all 5? One link leads to all 5
37943a03abb0454390c0e70fe51eae39.png09214c0c287f4549aaac5eb7ae87bc41.png
Fixed, is it better? Nothing extra?
User-agent: *
Allow: /
Disallow: /wp-admin
Disallow: */feed
Disallow: /cgi-bin
Disallow: /xmlrpc.php?rsd
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow : /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: /category/
Disallow: /author/
Disallow: /*?*
Disallow: /tag
Disallow: /js/
Disallow : /20*/*
Sitemap: andreyalexandrov.com/sitemap_index.xml
Host: andreyalexandrov.com
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: YandexImages
Allow: /wp-content/uploads/

Answer the question

In order to leave comments, you need to log in

3 answer(s)
P
Pavel Tupikov, 2015-09-17
@Traders

add a directive at the very end Host:: http://andreyalexandrov.com
With the sitemap everything is fine
and this is also possible
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: /*?*
Disallow: /tag
and at the very end of the custom rule
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: YandexImages
Allow: /wp-content/uploads/

R
ramjke, 2015-09-17
@ramjke

Disallow /wp-admin remove the slash at the end
Sitemap eats only one parameter, the bot eats only one Sitemap. Merge five documents into one or make one index file for several sitemaps.

E
Egor Nedbailo, 2015-09-17
@Negoro

You need to register separately for Yandex (add the Host directive :), Google and for everyone.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question