S
S
scdewt2017-07-19 18:38:06
PHP
scdewt, 2017-07-19 18:38:06

Redis server went away, why?

There is a really mysterious problem with redis, which periodically returns the Redis server went away error.
Given:
- 1 virtual server - 2 cores (IntelXeon E5-2670 processor), 8GB of RAM, debian 8
- the server is running Redis 3.2.9
- the load on Redis is extremely low:
8b85189d06294dad89c2c4dc592f4a7e.jpg
Periodically redis clients (php7, extension - https:// github.com/phpredis/phpredis) get the Redis server went away error . No error patterns were found.
Typically, clients get this error when trying to connect to Redis for the first time.
What I tried to do:
- disabled aof and rdb completely. I checked through the logs - snapshots stopped. But it did not help.
- then I tried to tune Redis itself:
-- maxmemory = 6Gb
-- timeout = 60
-- tcp-backlog=65000
-- maxclients = 100000
-- tcp-keepalive = 60
-- maxmemory-policy = allkeys-lru
- then tried to tune the system :
-- /proc/sys/fs/file-max = 817894
-- /proc/sys/net/core/somaxconn = 65000
-- vm.overcommit_memory=2
-- vm.overcommit_ratio = 100
-- net.ipv4.tcp_max_syn_backlog =5000
-- limits:
ad8c4a698b4d4ccb98dbe98e66154ac2.jpg
- then I decided to look at redis slowlog. I connected a slave and sent all heavy requests to it.
Currently slowlog is configured as slowlog-log-slower-than=5000 (5ms) and is empty. If you reduce it to 500 (0.5ms), then get\exists\smembers commands appear. (although exists should not be so long, but sometimes it gets into slowlog with values ​​of 600-1000 (0.6-1ms))
In the end, all this did not help.
From the nuances:
- now in php for redis the timeout is set to 1 second. With this timeout and with a constant load of the same type, redis can both work without errors for 1 hour, and throw errors at 2-10 per hour. If timeout is reduced to 0.5, then the number of errors increases. At the same time, the ping from the client to the redis server is 0.6-1 ms
- redis-benchmark shows a good picture:
88add4c592e440a9b8c3993a8db6fbd6.jpg
- I tried to write a script in php, which through fork creates 1000 clients with 500 incr commands each. And so almost always the script fulfills without problems, but periodically and it gives errors.
Where to dig further - I have no idea. Does anyone have any ideas or has anyone experienced something similar?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question