Web analytics software for high load?

Question

Analytics

Hello.
Faced such problem:
There is a highly loaded service (~200 requests / 1 sec).
- It is necessary to track user actions, while logging should occur entirely in the backend part (js inserts, as in Google Analytics, cannot be done).
- There should be an API to get statistics.
-Language - php.
-The user must be "tagged" with a unique cookie.
Currently used by Piwik . To put it mildly, he can’t cope with such a load, so he added query caching to Redis. With redis, requests are pulled by the php process in an endless loop (if there is data in the cache, 100 gets), then requests from users are emulated through curl. Roughly speaking, parallel requests are translated into serial ones. Everything would be fine, but I ran into the fact that 2 processes emulating requests cannot cope with the growth rate of the cache (~ 1.5 days, and the RAM will be busy). If you run more processes, the total load kills the server ((. Now I am rewriting the piwik logger to get away from http requests via curl.
Please recommend a more lightweight analogue that could cope with such a load.
Thanks in advance.

Answer 1

Do you want to watch real-time statistics? 200 rps are ridiculous numbers if you think a little about the architecture of this idea. We write logs in the correct format, we process them not in RT on a separate machine.

Answer 2

There is, for example, kontagent (http://www.kontagent.com/), it processes information on its servers, but the infection is expensive.

Answer 3

You can try using StatHat . This is not web analytics per se, but you can try to dump the processing on them by embedding charts in the internal system, rather than running to them. If the stage of leaving data to a third-party service is generally not an option in any form, then at least you can look at their architecture (starting from here ). Unfortunately, as I understand it, there are only separate pieces in opensource , in addition to client libraries

Answer 4

This, in my opinion, is completely perverted, but look at pinba - pinba.org In such loads, it will definitely cope and there is an opportunity to log your data. That's just the statistics on them will have to be processed by hand for the most part.