K
K
Kamil2019-09-24 18:47:51
Search Engine Optimization
Kamil, 2019-09-24 18:47:51

How to disable indexing of such links through robots?

Hello. Tell me please.
There is a Xenforo forum, connected to the POISON webmaster, I see that the bot is looking for pages with incorrect endings, it seems to me that this should not be done, how to limit such endings?
V2VnnG4Idajxo2.png

spoiler

User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Disallow: /tags/
Disallow: /search/
Disallow: /privacy-policy/
Disallow: */trophies/
Disallow: /lost-password/
Disallow: */help/
Allow: /
User-agent: Googlebot
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Disallow: /tags/
Disallow: /search/
Disallow: /privacy-policy/
Disallow: */trophies/
Disallow: /lost-password/
Disallow: */help/
Allow: /
User-agent: Yandex
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Disallow: /tags/
Disallow: /search/
Disallow: /privacy-policy/
Disallow: */trophies/
Disallow: /lost-password/
Disallow: */help/
Allow: /

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Arthur, 2019-09-24
@ar2rsoft

Disallow: *?*
Or

Disallow: *?order=*
Disallow: *?selected=*

V
Vladimir Kiper, 2019-09-25
@Kipeer

It’s better to write robots.txt on them, it doesn’t always save. And the bots still follow the links... if they exist. The question is what do you need it to index. To close from indexing is best! <meta name="robots" content="noindex, follow">
<meta name="robots" content="noindex, follow">

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question