S
S
sashakirichenko2019-09-17 10:12:27
Search Engine Optimization
sashakirichenko, 2019-09-17 10:12:27

Please rate the people, is robots.txt normal?

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/*.css*
Allow: /wp-content/*.js*
Allow: /wp-includes/*.js*
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-content/cache/
Disallow: /wp-trackback
Disallow: /wp-feed
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /wp-comments
Disallow: */trackback
Disallow: */comments
Disallow: /*.php
Disallow: /*?replytocom
Disallow: /*?turl

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Alexander Kuznetsov, 2019-09-17
@WSGR

Directives are valid.
But:
- the meaning of using * after .js/.css is not clear (even if it is necessary for some reason, the $ symbol would probably be more logical)
- the meaning of allowing these files to be indexed explicitly.
- the meaning of the prohibition of indexing some of the specified sections (for example, comments).
However, you know better what exactly you want to do.
What can be added:
- Sitemap
- The Host directive, but I draw your attention to the fact that it must be placed in a separate section exclusively for the Yandex bot (because, for example, it is not supported by Google), in practice it will probably parse normally, simply ignoring the line, but formally, may terminate processing on an invalid line.

R
Rookee, 2019-09-18
@Service_Rookee

Good afternoon!
We agree with the previous speaker that it makes no sense to put * after ".css" and ".js". Its absence also means any number of characters after. Yes, the "Sitemap: https://site.ru/sitemap.xml"
directive is missing . But Host can no longer be specified. Yandex stopped supporting this directive a year and a half ago. Now you can move your site right away with a 301 redirect.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question