T
T
trambi2018-06-12 11:46:10
API
trambi, 2018-06-12 11:46:10

How to protect rss from grabbers, but at the same time allow indexing for Yandex Turbo pages?

I want to connect turbo pages via rss in the Yandex webmaster, but my rss channel is closed from grabbers. How to make access to RSS only for Yandex turbo pages? WordPress.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
I
igorux, 2018-06-12
@trambi

Robots.txt is not specified for grabbers.
It is necessary to close access, and not hang a “Closed” sign.
As rightly noted above, at the server config level.
For example, for Apache, add .htaccess to the feed directory with something like this:
SetEnvIf User-Agent YandexBot/3.0 welcome=1
Order Allow,Deny
Deny from all
Allow from env=welcome
And allow Yandex to index in robots.

R
Roman Mirilaczvili, 2018-06-12
@2ord

Using the User-Agent, you can allow access to the resource only to those who are chosen at the level of the Apache .htaccess / Nginx web server configuration.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question