How to properly configure NGINX so as not to block search robots?

Question

Nginx

Hello!
I'm setting up Nginx to avoid primitive DDoS attacks, but I don't want to accidentally block crawlers. Now it's set up like this:

http {
  limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
  # ...
  server {
      location / {
          limit_req zone=one burst=20 nodelay;
          # ...
      }
  }
}

With the current configuration from 1 ip address there can be up to 10 requests per second with an interval of 100ms per request. At the same time, there can be a one-time "burst" of up to 20 simultaneous requests, after which each new request will also be processed with an interval of 100ms.

Is this configuration enough not to ban Google and Yandex search robots?

Answer 1

In robots.txtyou can specify limits on the frequency of requests - whoever does not take this into account is to blame.