I
I
Igor Potapov2018-11-27 20:58:33
htaccess
Igor Potapov, 2018-11-27 20:58:33

.htaccess settings - how to disable indexing from all bots?

Hello, tell me how to fill in .htaccess correctly?
The task is to prevent the bot from seeing my site
if you fill it out like this:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Googlebot) [NC]
RewriteRule .* - [R=403,L]
Everything works, you go under Google bot, then it does not see the site, but sees an error 403, but if you make a large list like
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Googlebot) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (google) [NC]
RewriteRule . * - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (HPWebPrintSmart) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (HPWeb PrintSmart) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (HP WebPrintSmart) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (HP Web PrintSmart) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AskJeeves) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Ask Jeeves) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (HTTrack) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Indy Library) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Mediapartners) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (RPT-HTTPClient) [NC]
RewriteRule .* - [R=403,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (ListChecker) [NC]
RewriteRule .* - [R=403,L]
or more, then error 500 crashes and the site does not work at all,
how to list all the bots correctly?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
O
Oleg, 2018-11-27
@Igor_criket

RewriteEngine On ONCE

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question