Answer the question
In order to leave comments, you need to log in
Will the PS follow the link if the page is in robots.txt?
The site has links to a page, for example /link.php
There are no rel="nofollow" links on the links, but there is a note in robots.txt Disallow: /link.php*
Will the search engine go to this page?
If so, how can you prevent the PS from visiting a certain page?
Answer the question
In order to leave comments, you need to log in
Forbid in any way, he wants to go anywhere. nofollow - forbids a link to an index. robots.txt - disables page indexing.
I asked a question once in Yandex:
Hello!
I have a page on the site that I do not want to see in the search, I closed it from indexing in robots.txt.
But this page is linked to by many other pages, both external and internal.
Questions:
1) If the number of links increases, will the robot at some point decide to add it to the index, or will it always be guided by the prohibition in robots.txt?
2) Do I need to surround links leading to a page closed in robots.txt with tags or is it not necessary or does it not play a role?
Answer:
1. Our robot does not index pages prohibited in robots.txt, neither in the absence of links to them, nor in the presence of any number of such.
2. It does not play any significant role. The only thing that is affected by links to pages that are forbidden to be crawled is the storage of information about these pages in the robot's database. In the absence of links, information about them is completely removed from the robot's database. If there are links, the robot remembers the pages in order to quickly index them if the ban is lifted.
If you want these links not to appear in the search results, then robots.txt is enough. It tells the search bot where it is not necessary to go, this file was created for this. It is clear that the bot can go there if he wants, this file is advisory in nature, but search engines like Google / Yandex follow the rules in robots.txt.
About rel="nofollow": it's not needed. I answered more about it recently in another question .
Therefore, the answer is this: the search engine will not visit this page, it will not be indexed.
Of course it will, it can't be banned. And what's the problem if the search robot goes there?
For Google:
robots.txt has note
Disallow: /link.php*
Will a search engine visit this page?
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question