K
K
Kaz522018-01-24 16:55:40
Google
Kaz52, 2018-01-24 16:55:40

What to prohibit in robots.txt?

There is a website hosted. Lies at /www/site.com/index.html
Questions.
What should be banned from indexing?
Before the www folder, there are folders like logs, tmp, etc. Should they be banned from indexing?
The site.com folder contains css, js, img, fonts folders and an .htaccess file. Should it be banned or not?

Answer the question

In order to leave comments, you need to log in

3 answer(s)
D
Dmitry Dart, 2018-01-24
@Kaz52

Before the www folder, there are folders like logs, tmp, etc. Should they be banned from indexing?

Not
No
You need to prohibit indexing directories with pages that you would not like to be indexed

P
Puma Thailand, 2018-01-24
@opium

None of this should be banned, how did you come up with such nonsense?

A
Andrey Kuskov, 2018-01-25
@Techinek

Usually, technical sections of the site, for example, the admin area, are prohibited from indexing. Through Ya.Webmaster in the indexing section, you can see which pages the robot finds and considers them to be of poor quality and of little use, and then it's up to you to decide whether to close them or modify them.
Search engines are already learning how to process styles and scripts, so there is no need to close them. A couple of years ago I shot a video on setting up robots.txt, I advise you to watch it. Will shed light on many moments: https://www.youtube.com/watch?v=xtxuWCzkyL0

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question