Answer the question
In order to leave comments, you need to log in
What to prohibit in robots.txt?
There is a website hosted. Lies at /www/site.com/index.html
Questions.
What should be banned from indexing?
Before the www folder, there are folders like logs, tmp, etc. Should they be banned from indexing?
The site.com folder contains css, js, img, fonts folders and an .htaccess file. Should it be banned or not?
Answer the question
In order to leave comments, you need to log in
Before the www folder, there are folders like logs, tmp, etc. Should they be banned from indexing?
None of this should be banned, how did you come up with such nonsense?
Usually, technical sections of the site, for example, the admin area, are prohibited from indexing. Through Ya.Webmaster in the indexing section, you can see which pages the robot finds and considers them to be of poor quality and of little use, and then it's up to you to decide whether to close them or modify them.
Search engines are already learning how to process styles and scripts, so there is no need to close them. A couple of years ago I shot a video on setting up robots.txt, I advise you to watch it. Will shed light on many moments: https://www.youtube.com/watch?v=xtxuWCzkyL0
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question