How to counteract site scraping?

Question

System administration

There is a popular real estate resource that contains many pictures of houses with descriptions. Recently, someone has been parsing the site and the traffic limits of the provider suffer from this. How can you resist parsing? In what direction to dig?
There are a whole bunch of IP-shnikov in the logs, it is not clear who to block.

Answer 1

To parse, you need to define some kind of template, how to search and by what criterion. Here is an option to complicate the task: captcha - to display some kind of information, ask for some kind of confirmation from the client. Break tags where it can be done painlessly, do not close them, implement the display of the same information in different ways. In general, add crap to parser writers) As soon as you notice that someone is parsing, change the page in detail.

Answer 2

change hosting to another with unlimited traffic.
And preferably not shared.

Answer 3

Set up a robots file. Maybe this is a good person and immediately the load will go away.

Answer 4

No way, for the most part. The only solution is to block ips that make too many requests per period using http_limit_req , for example.

Answer 5

You have not yet seen how search engines attack sites, in parallel with several IPs, they pull several pages, on shared hosting, load limits immediately work and block.

Answer 6

If pictures are parsed, then this is called hot-linking: here .
If the text is being parsed, then only by the behavioral filter should be filtered out by headers and IP / DNS.

Answer 7

as a joke - to remake the site under WebGL instead of images, cunningly cut textures, generally draw text with polygons, polish everything with encryption and a constant change in the algorithm - let them suffer.