V
V
Varoke2020-07-11 13:32:39
robots.txt
Varoke, 2020-07-11 13:32:39

How to disable bots for site indexing?

Is it possible to prevent site indexing from other sites / bots so as not to get backlinks from junk sites. Such as the top million sites in Alexarank and so on.

And how to make the site be able to index only specific bots, Google, Yandex, bing. For the rest, the site was closed.

Thanks for answers.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
Dimonchik, 2020-07-11
@dimonchik2013

User-agent: Googlebot
Disallow: /nogooglebot/
User-agent: *
Disallow: /
but in practice this is not enough and you still need to block the left IPs from the logs

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question