U
U
ufaloony2017-02-09 15:10:58
robots.txt
ufaloony, 2017-02-09 15:10:58

Can robots.txt and robots meta tag be used together?

Good afternoon!
There is a site in php, the pages of which are generated in the form site.ru/?view=razdel&id=1
I inserted robots meta tags on each page, somewhere I allowed indexing, somewhere I prohibited it. And I also created a robots.txt file, its contents are:
User-agent: *
Disallow: /images
Disallow: /js
Disallow: /lib
Disallow: /source
Disallow: /styles
Disallow: /tmpl
Disallow: /activate.php
Disallow: /login .php
Disallow: /settings.php
Host: site.ru
Do you need a robots.txt file at all?
I want to hide JavaScript in the js folder, classes in the lib folder, style files in the css folder, etc. index.php did not hide, or is it superfluous if there are meta tags?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
F
Fedor, 2017-02-10
@ufaloony

Priority on meta robots. That is, it is better to close through it. At the same time, if you close through the meta, you cannot register the page in the robots.
My robots.txt usually looks something like this:
User-agent: *
Disallow: /wp-*
Host: site.ru

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question