redis - many independent write servers -> 1 aggregating

Question

NoSQL

there are several combat servers,
each one writes some statistics to the radish,
each server has its own statistics

there is an aggregating server where you need to collect, group and calculate statistics,
and preferably in real time

while I see the options:
1 collect statistics from each radish on combat servers by timeout with a separate script, in php or python, and put into the aggregating radish
, but here there are many difficulties with ensuring the atomicity of operations

2 to do for each combat radish on the slave on the aggregating server and already on the server, by timeout, synchronize the slaves and group the data into the resulting database
, but here it arises the complexity of a lot of extra work

3 put statistics on combat servers in a queue in radish and pick them up with a remote php script in real time without grouping on radish, use radish only as a message manager,
but here there is a difficulty in large volumes of raw raw statistics transmitted over the network

thanks in advance

Answer 1

Isn't it an option to immediately send statistics when receiving a new "portion" of it? No external schedulers. Those. write directly to the aggregating radish, which is located on the local network, bypassing the memory of combat servers. Their role in the collection of statistics is not very clear. If the aggregator fails (memory is low, say), you can use a distributed redis connection (for example, implemented in the ruby ​​client, redis-rb), i.e. the server where the key data is located will be determined by the checksum of the key and thus the required performance will be achieved. The "aggregator" will be able to connect and get the correct data without a synchronization problem.