D
D
Dmitry Kulikov2015-04-20 14:37:47
Google
Dmitry Kulikov, 2015-04-20 14:37:47

Will the PS follow the link if the page is in robots.txt?

The site has links to a page, for example /link.php
There are no rel="nofollow" links on the links, but there is a note in robots.txt
Disallow: /link.php*
Will the search engine go to this page?
If so, how can you prevent the PS from visiting a certain page?

Answer the question

In order to leave comments, you need to log in

4 answer(s)
N
Nikolay, 2015-04-20
@dmitryi_k

Forbid in any way, he wants to go anywhere. nofollow - forbids a link to an index. robots.txt - disables page indexing.
I asked a question once in Yandex:
Hello!
I have a page on the site that I do not want to see in the search, I closed it from indexing in robots.txt.
But this page is linked to by many other pages, both external and internal.
Questions:
1) If the number of links increases, will the robot at some point decide to add it to the index, or will it always be guided by the prohibition in robots.txt?
2) Do I need to surround links leading to a page closed in robots.txt with tags or is it not necessary or does it not play a role?
Answer:
1. Our robot does not index pages prohibited in robots.txt, neither in the absence of links to them, nor in the presence of any number of such.
2. It does not play any significant role. The only thing that is affected by links to pages that are forbidden to be crawled is the storage of information about these pages in the robot's database. In the absence of links, information about them is completely removed from the robot's database. If there are links, the robot remembers the pages in order to quickly index them if the ban is lifted.

D
Dmitry Filimonov, 2015-04-20
@DmitryPhilimonov

If you want these links not to appear in the search results, then robots.txt is enough. It tells the search bot where it is not necessary to go, this file was created for this. It is clear that the bot can go there if he wants, this file is advisory in nature, but search engines like Google / Yandex follow the rules in robots.txt.
About rel="nofollow": it's not needed. I answered more about it recently in another question .
Therefore, the answer is this: the search engine will not visit this page, it will not be indexed.

U
un1t, 2015-04-20
@un1t

Of course it will, it can't be banned. And what's the problem if the search robot goes there?

A
aldous, 2015-04-30
@aldous

For Google:

robots.txt has note
Disallow: /link.php*
Will a search engine visit this page?

Materiel:
Robots.txt - config that controls access for the crawler. In Russian, the process is scanning.
In the example, access to the link is denied, this page will not be crawled. If it has been indexed once, then it will not be updated in the index anymore.
The answer is closer to no. This process looks a little different than just "goes in, does not go in".
The robots meta tag with the noindex parameter will completely disable both crawling and indexing, and exclude the page from search results.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question