A
A
Alexander Ivanov2017-02-09 14:25:59
Search Engine Optimization
Alexander Ivanov, 2017-02-09 14:25:59

How does robots work with subdomains?

*** It is also necessary to take into account that the site will have 2 copies (it will be opened from both forum.domain.ru and domain.ru/forum addresses), which may lead to the site being filtered by search engines for duplicating content. To prevent indexing by search engines of the technical address (domain.ru/forum), it is necessary to register in the robots.txt file a prohibition on indexing the specified directory (Disallow: / forum
/ ) site.ru, identical robots, will the robot continue to index dom.site.ru with the Disallow: /dom/ directive set? Content robots dom.site.ru

Host: site.ru
User-Agent: *
Disallow: /dom/ 
Sitemap: http://site.ru/sitemap.xml

Will the robots continue to index dom.site.ru, and how much do they depend on?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Andrey Nikolaev, 2017-02-09
@cimonlebedev

In general, if site.ru and dom.site.ru have the same robots.txt, then dom.site.ru will be an alias and should not be shown in the search (only site.ru will be), but this is just my guess

N
Neocaridina, 2017-02-10
@Neocaridina

Register in site.com/forum/.htaccess:

RewriteCond %{HTTP_HOST} ^site.com|^www.site.com [OR]
RewriteCond %{HTTP_HOST} ^www.dom.site.com [NC]
RewriteRule ^(.*)$ http://dom.site.com/$1 [R=301,L]

I do not advise you to rely on robots.txt in this regard.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question