U
U
unclechu2014-09-17 07:13:03
robots.txt
unclechu, 2014-09-17 07:13:03

How to ignore all domains except those specified in robots.txt?

In order to avoid getting into the index of developer forks of the project, as well as all the different domain variations like www. prefixes, you need to set a rule in robots.txt to ignore all domains except the specified one. How to do it?
And also how to make ignoring some directories also on this domain?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
M
Maxim Sergeev, 2014-09-17
@unclechu

If you mean under

all-different domain variations
- subdomains. Then you need to write in the root of each subdomain in robots.txt:
User-agent: *
Disallow: /
To disable the index of certain folders, use:
Disallow: /fork/
To specify the main host
Host: www.myhost.ru

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question