V
V
Vladimir Zp2017-03-25 21:27:34
System administration
Vladimir Zp, 2017-03-25 21:27:34

How to close a site for bots?

Good afternoon. Recently, my site has been visited in a sluggish mode by a small army of bots. It does not look like DDoS, since there are about 3 times more bots than people and this does not have a significant impact on the performance of the site itself.
The problem here is different. These bots randomly click on links, including ads, move around the site. Advertising platforms can either ban me for cheating, or stop working with me, since cheating is obvious, although I'm not doing it.
Also, these bots interfere with the analysis of the site - they blur behavioral factors, visit statistics and other metrics. The bot can be identified only through the web browser, it does not scroll the page and does not move the mouse, only once to randomly click on a link on the site. Visits 2-3 pages and leaves.
There are bots from all over the world. These are computers and smartphones. Use direct visits to various pages.
I didn’t think of anything better than how to close my site using iptables from all countries from which there were almost no visits before the invasion of bots. This allowed us to reduce the number of bots by several times, but they still exist. They used to break from all over the world, now from a limited part of it. But I want to get rid of them completely.
I'm just asking for advice on how to deal with such a scourge in my case. I will come up with the technical means myself. Now I just can't figure out which direction to take. Who and why does it - I have no idea. The site is not commercial, informational. I don't sell anything to anyone. This is just my blog with direct and adsense banner ads. Attendance of real people is 5000, with bots it used to be 20000, now it is somewhere around 1000-1500 bots per day.
At first I just waited, I thought the pests would get tired. But it doesn't seem to get boring. We must act somehow. I don’t really like the decision to ban entire countries, but I can’t think of anything better yet.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
X
xmoonlight, 2017-03-25
@xmoonlight

Don't worry - the algorithms of advertising platforms are smart enough to resist such things.
And in terms of damage to statistics - set up behavioral factors.

L
latteo, 2017-03-25
@latteo

it does not scroll the page and does not move the mouse

Write a small script that, on the first scroll or mouse movement event, sends an Ajax request to your server. Pre-google that with these events on smartphones. Same for click.
Then compare it with the statistics of visits and ban IPs that are not in the first array, but there are clicks.
You can further analyze the IP from the ban list and try to identify patterns. For example, they can all be in hoster ranges, i.e. this is clearly a proxy and the entire range can be added to the ban list.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question