D
D
Dubrovin2021-08-05 22:17:51
Nginx
Dubrovin, 2021-08-05 22:17:51

How to hide sitemap from public access via NGINX or Cloudflare?

Hello,
the site has a sitemap domain.com/sitemap_index.xml
, it is necessary that only search robots (Yandex, Google, Bing, Yahoo, Duckduckgo, etc.) have access to this sitemap and that it is closed to ordinary users.
Please tell me how can this be done through NGINX or using Cloudflare's WAF?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
K
ky0, 2021-08-05
@ky0

The difference between a search robot and a regular browser is only in the user agent, which is easy to fake. Of course, you can also add a check that IP addresses belong to search companies ... but why, in general? What is indexed by Google can be easily found in it.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question