T
T
thewizardplusplus2016-05-07 15:50:07
CMS
thewizardplusplus, 2016-05-07 15:50:07

Do I need to specify the admin area in robots.txt?

As far as I know, search engines index a site by following either internal links on it or external links to it. Thus, since the link to the admin panel is usually not indicated anywhere, the search bot will not be able to find it, and, accordingly, will not index it. Right? Or I'm wrong?
I'm just trying to write a robots.txt for a small blog and don't know what to put there. For, in theory, everything that a search bot can find should be indexed (posts, tags, pictures). And what should not, in theory, will not be found, since there are no references to it anywhere, neither external nor internal.
So is it necessary to specify the address of the admin area in robots.txt or does it make no sense?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
C
Cat Anton, 2016-05-07
@thewizardplusplus

Imagine this scenario: a link to the internal admin page of a site accidentally leaks into the network (there are millions of ways in the modern world for this to happen). And now the PS already know the URL. It suddenly turns out that, due to a bug, access to this page is open to everyone, including PS robots. You found a bug, fixed it, and your entire admin panel has already been indexed by the PS and saved in the cache by other bots. Do you need it? For business, write one line in robots.txt.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question