A
A
Alexander2020-08-15 00:23:38
Google
Alexander, 2020-08-15 00:23:38

How to force bots, including GoogleBot, not to visit non-existent pages of the site?

They did 301 redirects from old pages to new ones - all the same, Google and other systems remember these addresses and go to them a year later. Put 410 response. This will make it clear to the systems that it is necessary to no longer go to this address?
I don’t like it because it clogs the logs and makes the load on the site.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Anton Velichko, 2020-08-15
@sanok_ps

Google and other systems remember these addresses and go to them a year later.

There may be several reasons:
  1. old URLs are preserved in external links. If you can’t change them, then they will go.
  2. retained internal links to old URLs on your site. Check the internal linking, links on the site pages to non-existent pages may not be removed.

D
Dr. Bacon, 2020-08-15
@bacon

Put 410 response.
why? leave a redirect
I don’t like it because it clogs the logs and makes the load on the site.
Logs are needed for this, in order to know what is happening, complaining about "cluttering the logs" is generally stupid. And if this also causes a load, then it is better to spend time optimizing the work of the site code.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question