Are there ready-made scripts for protecting against parsers for nginx?

Question

Nginx

Greetings.
There is a busy site. Recently, competitors have appeared who parse it and immediately spread the information.
They are easily fired by the parser's ptr entry. Is there a ready-made script for nginx that analyzes the log, finds clients that look like bots, checks the ptr and adds it to the block?
Of course, I can write it myself, but I suspect that this has already been done. Google didn't come up with anything.

Answer 1

And then Gregor realized what had happened. These creatures could not put out the lamps: a hallucination cannot affect inanimate objects. He imagined that the room was getting darker and... He shot his own lamps! The subconscious fooled him again. Now Tenevik boldly stepped forward. Blaster was powerless.
Robert Sheckley, "Ghost 5"
Great plan, I think.
Competitors will fix bots in 5 minutes so that they are not caught, but a stupid bot will greatly complicate life for honest visitors and search engines. As a result, the site will die, and competitors will flourish.

Answer 2

1. Make a simple protection against parsing and "polish" SEO on the entire site as much as possible.
2. After creating a publication, immediately send a ping to search engines.
3. Open new articles to the public 10 minutes after the ping (just for ordinary browsers! For search engines - open immediately!).
4. For regular users of the community (excluding recently registered!) - open immediately after adding a publication.

Ping list

http://blogsearch.google.com/ping/RPC2
http://bblog.com/ping.php
http://bitacoras.net/ping
http://blog.goo.ne.jp/XMLRPC
http://blogdb.jp/xmlrpc
http://blogmatcher.com/u.php
http://bulkfeeds.net/rpc
http://coreblog.org/ping/
http://mod-pubsub.org/kn_apps/blogchatt
http://www.lasermemory.com/lsrpc/
http://ping.blo.gs/
http://ping.bloggers.jp/rpc/
http://ping.feedburner.com
http://ping.rootblog.com/rpc.php
http://pingoat.com/goat/RPC2
http://rpc.blogbuzzmachine.com/RPC2
http://rpc.blogrolling.com/pinger/
http://rpc.pingomatic.com
http://rpc.weblogs.com/RPC2
http://topicexchange.com/RPC2
http://trackback.bakeinu.jp/bakeping.php
http://www.bitacoles.net/ping.php
http://www.blogoole.com/ping/
http://www.blogpeople.net/servlet/weblogUpdates
http://www.blogshares.com/rpc.php
http://www.blogsnow.com/ping
http://www.blogstreet.com/xrbin/xmlrpc.cgi
http://www.mod-pubsub.org/kn_apps/blogchatter/ping.php
http://www.newsisfree.com/RPCCloud
http://www.newsisfree.com/xmlrpctest.php
http://www.snipsnap.org/RPC2
http://www.weblogues.com/RPC/
http://xmlrpc.blogg.de
http://1470.net/api/ping
http://api.feedster.com/ping
http://api.moreover.com/ping
http://api.moreover.com/RPC2
http://api.my.yahoo.com/RPC2
http://api.my.yahoo.com/rss/ping
http://blo.gs/ping.php
http://blogs.yandex.ru/
http://geourl.org/ping
http://ipings.com
http://ping.amagle.com/
http://ping.bitacoras.com
http://ping.blogmura.jp/rpc/
http://ping.blogs.yandex.ru/RPC2
http://ping.cocolog-nifty.com/xmlrpc
http://ping.exblog.jp/xmlrpc
http://ping.fc2.com/
http://ping.myblog.jp
http://ping.rss.drecom.jp/
http://ping.syndic8.com/xmlrpc.php
http://ping.weblogalot.com/rpc.php
http://ping.weblogs.se/
http://rcs.datashed.net/RPC2/
http://rpc.bloggerei.de/
http://rpc.bloggerei.de/ping/
http://rpc.copygator.com/ping/
http://rpc.icerocket.com:10080/
http://rpc.newsgator.com/
http://rpc.odiogo.com/ping/
http://rpc.technorati.com/rpc/ping
http://rpc.twingly.com
http://www.a2b.cc/setloc/bp.a2b
http://www.blogdigger.com/RPC2
http://www.blogoon.net/ping/
http://www.blogroots.com/tb_populi.blog?id=1
http://www.feedsubmitter.com
http://www.pingerati.net
http://www.pingmyblog.com
http://www.popdex.com/addsite.php
http://www.weblogalot.com/ping
http://xping.pubsub.com/ping
http://pingomatic.com
http://pingler.com
http://indexkings.com
http://totalping.com
http://pingfarm.com
http://pingmyurl.com
http://addurl.nu/
http://googleping.com
http://pingsitemap.com
http://pingbomb.com
http://mypagerank.net
http://twingly.com/ping
http://ping.in
http://coreblog.org/ping
http://feedshark.brainbliss.com
http://pingoat.net
http://backlinkping.com
http://nimtools.com/online-ping-website-tool
http://blogbuzzer.com
http://weblogs.com
http://pingmyblog.com
http://bulkping.com
http://auto-ping.com
http://rpc.weblogs.com
http://autopinger.com
http://icerocket.com
http://blogsnow.com/ping
http://weblogalot.com/ping
http://feedsubmitter.com
http://pingerati.net
http://pingmylink.com
http://syncr.com
http://blogpingtool.com
http://blogmatcher.com
http://pinggator.com
http://pingates.com

Answer 3

People simply do not bother, just parsing from their IP addresses. Tor is set up in 5 minutes. Buying a hundred proxies is also not a problem. There is no reliable protection against parsers. As soon as you block them by IP, they will quickly understand this and adapt.

Answer 4

An interesting topic, I opened Google, here's what I found, it can help:
https://moonback.ru/page/nginx-hotlink-protection
https://moonback.ru/page/site-protection
Here people advise using a log analyzer (fail2ban) - but IMHO this is nonsense!

Answer 5

If, as you write, bots fire on a pointer record, then use https://flant.ru/projects/nginx-http-rdns

Answer 6

Mono look towards fail2ban

Answer 7

Nginx Bad Bot and User-Agent Blocker, Spam Referrer Blocker, Anti DDOS, Bad IP Blocker and Wordpress Theme Detector Blocker:
https://github.com/mitchellkrogza/nginx-ultimate-b...
You can also use Fail2ban