D
D
DenimTornado2015-08-26 22:15:18
Search Engine Optimization
DenimTornado, 2015-08-26 22:15:18

What do the different user-agents in robots.txt mean?

Good afternoon!
There is robots.txt:

User-Agent: Yandex
Disallow: /doc
Disallow: /promo
Disallow: /dev
Disallow: /client
Disallow: /wiki
Disallow: /statistics
Disallow: /statistics_client
Disallow: /dima
Disallow: /testovyi
Disallow: /*search.php
Disallow: *sphrase_id=*
Disallow: *openstat=*
Disallow: *from=*
Disallow: *gclid=*
Disallow: *utm*
Disallow: *yclid=*
Disallow: /*forgot_password
Disallow: /*register
Disallow: /*change_password
Disallow: *offset=*
Disallow: *CODE=*
Disallow: /*logo.swf
Disallow: /*.js
Disallow: /new
Disallow: *bitrix_include
Disallow: *bitrix/spread.php
Disallow: *clear_cache
Disallow: /samostoyatelno/?*
Disallow: /samostoyatelno/*?*
Disallow: /samostoyatelno/*/?
Disallow: /samostoyatelno/*/*?*
Disallow: /samostoyatelno/*/*/*?*
Disallow: *SECTION_CODE=*
Disallow: *otp.php*
Disallow: *admin*
Disallow: *login=*
Disallow: *seoorder.php*
Disallow: /?%ECx
Disallow: /poiskovoe_prodvizhenie_sajtov/trafik/?*
Disallow: /studio/clients/old/
Disallow: /studio/bill/$
Disallow: /mobileform.php
Disallow: *feedback.php*
Disallow: *template=*
Disallow: /pixelplus-2015.php
Clean-param: CODE /studio/stat/
Host: www.pixelplus.ru

User-Agent: *
Disallow: /doc
Disallow: /promo
Disallow: /dev
Disallow: /client
Disallow: /wiki
Disallow: /statistics
Disallow: /statistics_client
Disallow: /dima
Disallow: /testovyi
Disallow: /*search.php
Disallow: *sphrase_id=*
Disallow: *openstat=*
Disallow: *from=*
Disallow: *gclid=*
Disallow: *utm*
Disallow: *yclid=*
Disallow: /*forgot_password
Disallow: /*register
Disallow: /*change_password
Disallow: *offset=*
Disallow: *CODE=*
Disallow: /*logo.swf
Disallow: /*.js
Disallow: /new
Disallow: *bitrix_include
Disallow: *bitrix/spread.php
Disallow: *clear_cache
Disallow: /samostoyatelno/?*
Disallow: /samostoyatelno/*?*
Disallow: /samostoyatelno/*/?
Disallow: /samostoyatelno/*/*?*
Disallow: /samostoyatelno/*/*/*?*
Disallow: *SECTION_CODE=*
Disallow: *otp.php*
Disallow: *admin*
Disallow: *login=*
Disallow: *seoorder.php*
Disallow: /?%ECx
Disallow: /poiskovoe_prodvizhenie_sajtov/trafik/?*
Disallow: /studio/clients/old/
Disallow: /studio/bill/$
Disallow: /mobileform.php
Disallow: *feedback.php*
Disallow: *template=*
Disallow: /pixelplus-2015.php

I can’t understand why all the directives are duplicated separately for Yandex as well?
Thanks in advance!

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question