Weird bots with javascript enabled

Question

Bots

Good evening, habravchane.

Since September 2, I have been observing an interesting picture in my project: bots are constantly walking around the site, 1-2 requests per second (100k+ views per day), judging by the analytics. Features of these bots:
- Ignore robots.txt.
- Presented exclusively as IE6-8.
- They behave like a full-fledged browser: they execute javascript, accept cookies, etc.
- They open exactly 2 pages per session (then reset the cookie?), follow links from the main page.
- Do not submit (do not even try) forms.
- Geographically, calls come from all over the world, but most of them are from NY.

They do not create any tangible load, but they spoil the analytics. Yes, and I wonder what it is and who needs it? Previously somehow did not come across.
Can you tell me?

UPD: I looked at the logs on the server, the bots make about 5-6 requests per second. I did not find anything unusual in the headlines to safely weed out without hurting users.

Answer 1

Interesting topic.
Maybe some indicators wind up, for example, transitions from search engines? Maybe toolbars are installed in these IE and they wind up the numbers in them? Maybe they check your site for load?
You can try to catch the bot by making an invisible (via CSS) link off the edge of the screen. A human won't click on it, but a bot crawling through HTML code or parsing a DOM tree will get caught. You can try to listen to the mousemove event - a live person moves the mouse, unlike a bot.
Also, maybe it's worth measuring the location of the window on the screen for these bots? (for example, as described here: stackoverflow.com/a/504097 ). After all, if this is a real IE controlled by a virus, then most likely the window is pushed over the edge of the screen so as not to interfere with the user.

Answer 2

Yes, I've noticed the same thing myself.
I was able to cut down some of the robots using this service: www.stopforumspam.com
An example of accessing the API: www.stopforumspam.com/api?f=json&unix&confidence&ip=8.8.8.8
True, there is a limit of up to 20,000 requests per day. If you use bulk requests, you can check up to 100,000 IPs per day. I have enough :)
Of the shortcomings - they work with individual IPs. When I look through the logs “by hand”, it is quite obvious that some requests come from the same subnet and are formed by the same robots. But the service does not take this into account. To finish it, analyzing subnets, hands have not reached yet ...

Answer 3

some bots don't want to load pictures. Try to see if yours are loading.

Answer 4

They can do a lot. Let's say parse you if it's not an option without js, or it's too messy, or they can, say, click on contest ads.

Answer 5

Whose AS? Are these manual servers or zombie workstations?
In the order of delirium - maybe they train zombies? those. as an option, walking bots on normal sites in order to fill in the statistics of visiting different sites by these bots, so that search engines perceive bots more believably.
Another option is advertising clicks to get a ban from advertising networks ...
Try to check for yourself how well their JS works. Maybe they just deceive the statistics check?
As an option, test bots.

Answer 6

Faced the same problem - 80% of my traffic is bots. They may be from different countries, but mostly the USA. They can be direct, or they can come from Google. But this is what unites them:
1. The mouse does not move in the webviewer.
2. You might think that people with a tablet, but the page does not scroll throughout the entire session.
3. The bot comes to one of the pages, opens another one and the session ends.
4. The average duration of their sessions is 20-30 seconds.
How can I get them off the site?