X
X
XelaBender2022-02-04 18:58:29
Redis
XelaBender, 2022-02-04 18:58:29

Why does Redis server crash "Out Of Memory allocating 16400 bytes!"?

Dedicated server
CPU: Intel Xeon-E 2388G - 8c/16t - 3.2 GHz/4.6 GHz
Memory: DDR4 64Gb
Disk: 1.92 TB SSD NVMe
System: Ubuntu Server 20.04 "Focal Fossa"

spoiler

Out Of Memory allocating 16400 bytes!

=== REDIS BUG REPORT START: Cut & paste starting from here ===
735991:M 04 Feb 2022 12:05:24.303 # ------------------------------------------------
735991:M 04 Feb 2022 12:05:24.303 # !!! Software Failure. Press left mouse button to continue
735991:M 04 Feb 2022 12:05:24.303 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.c:3925
735991:M 04 Feb 2022 12:05:24.303 # (forcing SIGSEGV in order to print the stack trace)
735991:M 04 Feb 2022 12:05:24.303 # ------------------------------------------------
735991:M 04 Feb 2022 12:05:24.303 # Redis 5.0.7 crashed by signal: 11
735991:M 04 Feb 2022 12:05:24.303 # Crashed running the instruction at: 0x5573e0abf1d8
735991:M 04 Feb 2022 12:05:24.303 # Accessing address: 0xffffffffffffffff
735991:M 04 Feb 2022 12:05:24.303 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/bin/redis-server 127.0.0.1:6379(_serverPanic+0x148)[0x5573e0abf1d8]

Backtrace:
/usr/bin/redis-server 127.0.0.1:6379(logStackTrace+0x4f)[0x5573e0ac13af]
/usr/bin/redis-server 127.0.0.1:6379(sigsegvHandler+0xb5)[0x5573e0ac1bb5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fd878a593c0]
/usr/bin/redis-server 127.0.0.1:6379(_serverPanic+0x148)[0x5573e0abf1d8]
/usr/bin/redis-server 127.0.0.1:6379(redisOutOfMemoryHandler+0x3a)[0x5573e0a7733a]
/usr/bin/redis-server 127.0.0.1:6379(zmalloc+0x47)[0x5573e0a81227]
/usr/bin/redis-server 127.0.0.1:6379(+0x406a1)[0x5573e0a886a1]
/usr/bin/redis-server 127.0.0.1:6379(addReplyDouble+0xe4)[0x5573e0a89ef4]
/usr/bin/redis-server 127.0.0.1:6379(zrangeGenericCommand+0x51e)[0x5573e0aaaefe]
/usr/bin/redis-server 127.0.0.1:6379(call+0xaf)[0x5573e0a79b2f]
/usr/bin/redis-server 127.0.0.1:6379(execCommand+0xa0)[0x5573e0abe4c0]
/usr/bin/redis-server 127.0.0.1:6379(call+0xaf)[0x5573e0a79b2f]
/usr/bin/redis-server 127.0.0.1:6379(processCommand+0x556)[0x5573e0a7a436]
/usr/bin/redis-server 127.0.0.1:6379(processInputBuffer+0x171)[0x5573e0a8b0a1]
/usr/bin/redis-server 127.0.0.1:6379(aeProcessEvents+0x112)[0x5573e0a732d2]
/usr/bin/redis-server 127.0.0.1:6379(aeMain+0x2b)[0x5573e0a736fb]
/usr/bin/redis-server 127.0.0.1:6379(main+0x4de)[0x5573e0a6dade]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fd8788790b3]
/usr/bin/redis-server 127.0.0.1:6379(_start+0x2e)[0x5573e0a6dd3e]

------ INFO OUTPUT ------
# Server
redis_version:5.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:636cde3b5c7a3923
redis_mode:standalone
os:Linux 5.4.0-91-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:9.2.1
process_id:735991
run_id:2d1c52d190d781a088b5c5f2fffc11bddfbfec14
tcp_port:6379
uptime_in_seconds:387322
uptime_in_days:4
hz:10
configured_hz:10
lru_clock:16586372
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf

# Clients
connected_clients:12
client_recent_max_input_buffer:2
client_recent_max_output_buffer:57739264
blocked_clients:0

# Memory
used_memory:914899080
used_memory_human:872.52M
used_memory_rss:988934144
used_memory_rss_human:943.12M
used_memory_peak:1018331200
used_memory_peak_human:971.16M
used_memory_peak_perc:89.84%
used_memory_overhead:4830596
used_memory_startup:796240
used_memory_dataset:910068484
used_memory_dataset_perc:99.56%
allocator_allocated:911523528
allocator_active:921133056
allocator_resident:989069312
total_system_memory:67335008256
total_system_memory_human:62.71G
used_memory_lua:41984
used_memory_lua_human:41.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:9609528
allocator_rss_ratio:1.07
allocator_rss_bytes:67936256
rss_overhead_ratio:1.00
rss_overhead_bytes:-135168
mem_fragmentation_ratio:1.09
mem_fragmentation_bytes:77643448
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:3967564
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:11618
rdb_bgsave_in_progress:0
rdb_last_save_time:1643976309
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:4
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:67022848
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:28
total_commands_processed:243415647
instantaneous_ops_per_sec:606
total_net_input_bytes:21289486937
total_net_output_bytes:7955222927584
instantaneous_input_kbps:51.94
instantaneous_output_kbps:7.75
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:13
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:2308838
keyspace_misses:562996
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:51304
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:00fec0c5f2f1d8b03259a55e70f9d116be31d893
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:5862.124498
used_cpu_user:25935.504398
used_cpu_sys_children:1969.233247
used_cpu_user_children:21740.043277

# Commandstats
cmdstat_expire:calls=18216855,usec=12486181,usec_per_call=0.69
cmdstat_scan:calls=5183,usec=1698273,usec_per_call=327.66
cmdstat_zremrangebyrank:calls=66,usec=20,usec_per_call=0.30
cmdstat_zrem:calls=1229,usec=868,usec_per_call=0.71
cmdstat_del:calls=63,usec=82,usec_per_call=1.30
cmdstat_zremrangebyscore:calls=18727742,usec=261476333,usec_per_call=13.96
cmdstat_lpush:calls=18216845,usec=71587895,usec_per_call=3.93
cmdstat_smembers:calls=12896,usec=24079,usec_per_call=1.87
cmdstat_ltrim:calls=18216845,usec=44053712,usec_per_call=2.42
cmdstat_unwatch:calls=19061039,usec=4475704,usec_per_call=0.23
cmdstat_zcard:calls=224529,usec=72024,usec_per_call=0.32
cmdstat_setnx:calls=33,usec=119,usec_per_call=3.61
cmdstat_llen:calls=74843,usec=91836,usec_per_call=1.23
cmdstat_zrangebyscore:calls=648,usec=2175,usec_per_call=3.36
cmdstat_zrevrange:calls=1419118,usec=87614795,usec_per_call=61.74
cmdstat_watch:calls=10,usec=64,usec_per_call=6.40
cmdstat_exec:calls=19061029,usec=24857149394,usec_per_call=1304.08
cmdstat_command:calls=1,usec=546,usec_per_call=546.00
cmdstat_config:calls=4,usec=42,usec_per_call=10.50
cmdstat_zrange:calls=311881,usec=17758756211,usec_per_call=56940.81
cmdstat_lrange:calls=84238,usec=6204684013,usec_per_call=73656.59
cmdstat_info:calls=3,usec=618,usec_per_call=206.00
cmdstat_bgsave:calls=1,usec=48814,usec_per_call=48814.00
cmdstat_multi:calls=19061030,usec=5393238,usec_per_call=0.28
cmdstat_hset:calls=18731519,usec=40855589,usec_per_call=2.18
cmdstat_hsetnx:calls=1176,usec=1936,usec_per_call=1.65
cmdstat_zincrby:calls=10,usec=109,usec_per_call=10.90
cmdstat_exists:calls=84228,usec=186379,usec_per_call=2.21
cmdstat_hincrby:calls=36436402,usec=116822029,usec_per_call=3.21
cmdstat_ping:calls=6,usec=2,usec_per_call=0.33
cmdstat_select:calls=24,usec=5,usec_per_call=0.21
cmdstat_hdel:calls=10,usec=11,usec_per_call=1.10
cmdstat_hgetall:calls=416034,usec=2142105,usec_per_call=5.15
cmdstat_get:calls=1,usec=1,usec_per_call=1.00
cmdstat_client:calls=3,usec=8,usec_per_call=2.67
cmdstat_zadd:calls=54806686,usec=559082513,usec_per_call=10.20
cmdstat_hget:calls=243417,usec=350278,usec_per_call=1.44

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=657,expires=0,avg_ttl=0
db1:keys=533,expires=73,avg_ttl=138730199

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vitaly Karasik, 2022-02-06
@vitaly_il1

I advise you to install the new version - 6.2 ( https://redis.io/download ). Or at least 5.0.14 if it's really important to be with 5.x.
5.0.7 was released over two years ago.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question