How to disable access to url search robots?

Question

Google

Hello, in fact, the question is described in the subject, here is the robots.txt:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /wp-content/
Disallow: /tag/
Disallow: /category/
Disallow: /archive/
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=
Disallow: /?s=
Disallow: /%D1%81%D0%B5%D0%BA%D1%80%D0%B5%D1%82%D1%8B-%D1%81%D1%82%D0%B8%D0%BB%D1%8C%D0%BD%D0%BE%D0%B9-%D1%81%D0%B2%D0%B0%D0%B4%D1%8C%D0%B1%D1%8B/

Interested in the last link, is it prohibited? I check like this:

wget --user-agent="Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots" http://mydomain.com/%D1%81%D0%B5%D0%BA%D1%80%D0%B5%D1%82%D1%8B-%D1%81%D1%82%D0%B8%D0%BB%D1%8C%D0%BD%D0%BE%D0%B9-%D1%81%D0%B2%D0%B0%D0%B4%D1%8C%D0%B1%D1%8B

As a result, the content of the page is downloaded, but it is, in theory, closed in robots.txt
Or so I won’t check, because the search engines themselves read robots.txt first?

Answer 1

The robots file is a wish .
This is your wish, which decent search engines fulfill.
Do it voluntarily.
But this file does not force them to anything and does not restrict them in any way.
If they wish, they can ignore the contents of this file, although this is bad form.
VGetu naturally does not care about this rule. VGet executes your command. What the man told him, he did.

Answer 2

To check if robots.txt is configured correctly, you can use search engine tools. Yandex has https://webmaster.yandex.ru/robots.xml. Google has https://www.google.com/webmasters/tools/robots-tes... , but you can only check with a verified site