How to determine the fact of site parsing?

Question

Parsing

Greetings
Sometimes some people parse the site, articles from it, some even manage to parse the design. Is it possible to somehow track the parsing of a site at the program level and, for example, block the parsing process? Maybe someone has good advice or a good bookmarked article.

Answer 1

No. A well-functioning parser cannot be distinguished from a regular user(s).

Answer 2

Limit the number of hits from 1 IP for a certain period of time.

Answer 3

Search bots also crawl websites.
If we are talking about a store, then there will always be parsing and tyrating.
If these are articles, then before publishing them on the site, you can first show them to Yandex so that it knows where the article first appeared.
The single request per IP limit is easily bypassed.

Answer 4

I'm thinking about this problem myself right now.
1) Blocking by ip in the first place
2) Dynamic content. I haven’t figured out how yet, but I need to somehow change the structure and block classes with each request. The parser is primarily focused on the structure of the page.
3) you can load important content with ajax
4) you can make important content with pictures. Like Avito, they show phones with pictures.

Answer 5

it’s impossible for articles,
there are ways for Avito, but overheads for the process - for example, Googlebots and others are easily determined