Answer the question
In order to leave comments, you need to log in
V
V
vox_termin2017-04-13 13:12:25
vox_termin, 2017-04-13 13:12:25
Is it possible to give different feeds for different robots in robots.txt?
Good afternoon!
I'm interested in the question, is it possible to give one feed to a Yandex bot, and give the other feed to a Google or Rambler bot?
2 answer(s)
@AXP-dev
@secsite
Robots does not serve for permissions/prohibitions to "give away", but for asking (!) to exclude (!!) from the search results (not to be confused with indexing - they index everything anyway :) ). But whether the bots obey or not is their business.
www.robotstxt.org/robotstxt.html
A
Alexander Pushkarev, 2017-04-13 @AXP-dev
All in one robots.txt
User-agent: Yandex
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: StackRambler
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /S
Site Developer, 2017-04-13 @secsite
give away
Robots does not serve for permissions/prohibitions to "give away", but for asking (!) to exclude (!!) from the search results (not to be confused with indexing - they index everything anyway :) ). But whether the bots obey or not is their business.
www.robotstxt.org/robotstxt.html
Similar questions
I
Z
G
F
Faflick2021-09-10 11:52:36
How to close all pages after a certain level of nesting from indexing?
1Reply
A
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question