A
A
agromov912020-02-18 13:26:09
htaccess
agromov91, 2020-02-18 13:26:09

Bot redirect via htaccess?

How can I block all bots except Yandex and Google?
Or send to a separate page.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Alexander Dokin, 2020-02-18
@alnidok

In a nutshell, see below for more details.
To prevent crawlers from crawling, robots.txt is usually used,

like this
User-agent: Googlebot
Allow: /

User-agent: Yandex
Allow: /

User-agent: *
Disallow: /

Detailed lists of user agents: google and yandex .
But robots.txt is just a warning, so some SEO bots will simply ignore it.
Now about .htaccess: when processing requests, the web server can only focus on what is sent to it in the headers. Any bot can send anything to the User-agent, so it will not be possible to distinguish them from a regular user.
Yes, there are projects like this one that update the bases for defining "bad bots". If that's what you really need, check it out.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question