J
J
John Ship2021-08-23 12:13:38
Google
John Ship, 2021-08-23 12:13:38

Clean-param parameter in robots.txt. How to make friends with Google and Yandex at the same time?

The fact that the Clean-param parameter is not perceived by Google has already been written more than once (and on Yandex itself too). With this, everything is clear. But what if I want to use robots.txt for both google AND Yandex at the same time (and indeed any other PS)?
It seems that there is a directive that divides the robots.txt file into sections, allowing you to set a parameter exclusively for Yandex. But when checking with a utility from Google, it generates an error (Yandex considers this file to be correct)

6123646d27d16574627797.png

: .

Of course, it is possible that I will define a bot on the server that requests a file and give it different options, but this is very inconvenient, and you can’t write rules for all bots. So, for example, the SemRush service also does not accept such a file. And I think most other services are the same, because this parameter is accepted only by Yandex.

Does anyone use the Clean-param parameter on their sites and use both Google Webmaster Tools AND Yandex.Webmaster at the same time? How do you deal with errors (in Yandex) of duplicate pages with "unnecessary" parameters?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
svm, 2021-08-26
@svm

The tool writes that it does not recognize the syntax of this directive, while others will be taken into account.
Those. from using this directive, other rules do not break. You can also verify this in this tool by inserting any url that falls under other rules.
I had this directive installed for a long time, but I don't remember whether I checked it or not in this tool. In any case, no warnings were displayed in the GSC.
At the moment, I don’t use this directive, because I switched to using the laravel framework and wrote an intermediary in it that removes unnecessary parameters, but leaves the necessary ones.
As an untested option in practice, I can suggest this option:
Disallow: /*?*amp
Disallow: /*?*__cf_chl
I just came up with it and quickly tested it. It seems to work.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question