Answer the question
In order to leave comments, you need to log in
A
A
agromov912020-02-18 13:26:09
agromov91, 2020-02-18 13:26:09
Bot redirect via htaccess?
How can I block all bots except Yandex and Google?
Or send to a separate page.
1 answer(s)
@alnidok
Detailed lists of user agents: google and yandex .
But robots.txt is just a warning, so some SEO bots will simply ignore it.
Now about .htaccess: when processing requests, the web server can only focus on what is sent to it in the headers. Any bot can send anything to the User-agent, so it will not be possible to distinguish them from a regular user.
Yes, there are projects like this one that update the bases for defining "bad bots". If that's what you really need, check it out.
A
Alexander Dokin, 2020-02-18 @alnidok
In a nutshell, see below for more details.
To prevent crawlers from crawling, robots.txt is usually used,
like this
User-agent: Googlebot
Allow: /
User-agent: Yandex
Allow: /
User-agent: *
Disallow: /Detailed lists of user agents: google and yandex .
But robots.txt is just a warning, so some SEO bots will simply ignore it.
Now about .htaccess: when processing requests, the web server can only focus on what is sent to it in the headers. Any bot can send anything to the User-agent, so it will not be possible to distinguish them from a regular user.
Yes, there are projects like this one that update the bases for defining "bad bots". If that's what you really need, check it out.
Similar questions
M
O
M
V
R
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question