W
W
wowwek2018-03-12 22:04:59
Search Engine Optimization
wowwek, 2018-03-12 22:04:59

How to correctly configure robots.txt by regular expression?

What I have: in the webmaster on the "Bypass Pages" tab:

  • /index/0-31-0-31 200 OK
  • /index/0-33-0-33 200 OK
  • /index/0-34-0-34 200 OK
  • /index/0-69-0-69 200 OK
How can you remove such pages regularly?
The site is spinning on yukoz, recently the pages in the issue have sagged a lot, I suspect that this is the case.
Does not help:
  • Disallow: /*-*-*-*-987$
  • Disallow: /*0-*-0-17$
  • Disallow: /*-0-0-

Answer the question

In order to leave comments, you need to log in

1 answer(s)
M
Maxim Timofeev, 2018-03-12
@wowwek

I would ask myself the question "where did Yasha get these links", because somewhere they are on the site and this is the problem. If there are no links from public pages, the search engine will not index them.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question