A
A
Andrnofollow2020-12-23 11:15:40
Search Engine Optimization
Andrnofollow, 2020-12-23 11:15:40

How to close all internal pages of a folder from scanning?

The site category has a large number of pages that I don't need. In this case, the category itself must be in the index. How to format it correctly through robots.txt?
Now I wrote it like this, but I don't know if it's right:
Disallow: /category
Allow: /category/$

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Alexander Denisov, 2020-12-23
@Grinvind

Does your category page end with a slash?
If yes, then I would correct:

Disallow: /category/
Allow: /category/$

To check, use the "Check robots.txt" tool in Yandex.Webmaster. GSC also has a similar tool.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question