How to organize deferred processing of statistics?

Question

Redis

Hello.
There is a simple system for recording link click statistics. The frontend, after redirecting, RPUSH sends the click data (LinkID, session, timestamp) to a key named "clicks_queue". Every 5 seconds, the worker wakes up and uses the LRANGE clicks_queue 0 -1 command to extract all clicks from the queue and writes changes grouped by links to the database. With small loads, everything is OK, but as soon as the load increases, there is a discrepancy in the logs and statistics. I suspect that while the worker is reading data from the queue, the frontend adds more clicks. But when the worker has retrieved all the elements, it clears the ltrim'om queue and the data that has been added is also deleted. What am I doing wrong, and what is the best way to organize deferred processing of statistics?
Thanks in advance.

Answer 1

look towards the queue server,
I don’t know how RPUSH works, but in Python, this relies on zeromq or gearman at worst

Answer 2

Pull data and clear the list inside a transaction:
redis.io/topics/transactions

MULTI
LRANGE clicks_queue 0 -1
DEL clicks_queue
EXEC