S
S
slonikkk2015-04-21 08:28:26
Search Engine Optimization
slonikkk, 2015-04-21 08:28:26

Correct robots.txt for wordpress?

Good afternoon.
NOT when I didn’t think that finding robots for WordPress was such a problem. Rummaged through many blogs and everywhere everything is different, knowledge in this area. Who can share a robots file that actually works? I need to close everything superfluous, leave only pages, posts, and images.
Made 301 redirects.
But I also think that the robot should close everything in the file, but what exactly?
htaccess itself
RewriteCond %{QUERY_STRING} ^replytocom= [NC]
RewriteRule (.*) $1? [R=301,L]
RewriteRule (.+)/feed /$1 [R=301,L]
RewriteRule (.+)/attachment /$1 [R=301,L]
RewriteRule (.+)/comment-page /$1 [R=301,L]
RewriteRule (.+)/comments /$1 [R=301,L]
RewriteRule (.+)/trackback /$1 [R=301,L]
RewriteCond %{QUERY_STRING} ^attachment_id= [NC]
RewriteRule (.*) $1? [R=301,L]
While the robot looks like this:
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-admin
Disallow: /wp- includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-feed
User-agent: Yandex
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-feed
Host: site
Sitemap: site/sitemap.xml
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: YandexImages
Allow: /wp-content/uploads/
What else to add to the robot ?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
dadster, 2015-05-21
@dadster

and here is how the notorious Yoast.com do it:
User-Agent: *
Disallow: /out/
Simple and tasteful)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question