KeyDB icon indicating copy to clipboard operation
KeyDB copied to clipboard
verified publisher icon Snapchat

[CRASH] KeyDB 6.3.4 crashed by signal: 11, si_code: 1

Open broken-ufa opened this issue 1 year ago • 4 comments

Crash report

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
3907654:3907709:M 10 Sep 2024 17:43:09.802 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
3907654:3907709:M 10 Sep 2024 17:43:09.802 # Accessing address: 0x1
3907654:3907709:M 10 Sep 2024 17:43:09.802 # Crashed running the instruction at: 0x62f7bf5c4068

------ STACK TRACE ------
EIP:
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x62f7bf5c4068]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x704782042520]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x62f7bf5c4068]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](+0x1d196a) [0x62f7bf5b096a]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](readQueryFromClient(connection*)+0x39c) [0x62f7bf5a2d0c]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x1e0) [0x62f7bf6986e0]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](ProcessEventCore+0xf8) [0x62f7bf56f338]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeProcessEvents+0x17d) [0x62f7bf5733fd]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeMain+0x3e) [0x62f7bf573a6e]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](workerThreadMain(void*)+0xfe) [0x62f7bf58be9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x704782094ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x704782126850]

------ REGISTERS ------
3907654:3907709:M 10 Sep 2024 17:43:09.810 #
RAX:4297ccfb0fc91ac9 RBX:00007047812b7380
RCX:0000704583ce43d3 RDX:0000000000000001
RDI:0000000000c91ac9 RSI:0000000000000000
RBP:00007046b85c4163 RSP:000070475affc6e0
R8 :845d0f850ffe5bcb R9 :0000704626f6f3bb
R10:0000000000000003 R11:000062f7bfb0a010
R12:0000704655081b90 R13:00000000003ba078
R14:00007045fea9f800 R15:0000704655081b90
RIP:000062f7bf5c4068 EFL:0000000000010202
CSGSFS:002b000000000033
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ef) -> 0000704700001fba
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ee) -> 0000704700000000
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ed) -> 000070475affd978
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ec) -> 0000000059b0b680
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6eb) -> 0100010100000001
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ea) -> 000070475affc74c
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e9) -> 000000005affc838
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e8) -> 000000000000001f
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e7) -> 000062f7bf58493c
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e6) -> 0000000000000000
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e5) -> 0000000000000001
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e4) -> 000062f7bfc4b920
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e3) -> 000070447f4cd740
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e2) -> 000070447e993ac0
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e1) -> 000062f7bfc49c00
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e0) -> 000070475affdb38

------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b532cd0401cb0da4
redis_mode:cluster
os:Linux 6.5.0-27-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:3907654
process_supervised:no
run_id:3025d938de37fc657b06ca71bd393385236bc727
tcp_port:7000
server_time_usec:1725990189810847
uptime_in_seconds:427643
uptime_in_days:4
hz:200
configured_hz:200
lru_clock:14714157
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb-master2-keydb2.conf
availability_zone:
features:cluster_mget

# Clients
connected_clients:3020
cluster_connections:16
maxclients:10000
client_recent_max_input_buffer:81968
client_recent_max_output_buffer:5369
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:5
thread_0_clients:526
thread_1_clients:441
thread_2_clients:521
thread_3_clients:514
thread_4_clients:535
thread_5_clients:485

# Memory
used_memory:8545836632
used_memory_human:7.96G
used_memory_rss:9131364352
used_memory_rss_human:8.50G
used_memory_peak:9357867144
used_memory_peak_human:8.72G
used_memory_peak_perc:91.32%
used_memory_overhead:1068160545
used_memory_startup:5103768
used_memory_dataset:7477676079
used_memory_dataset_perc:87.55%
allocator_allocated:8549300992
allocator_active:9017274368
allocator_resident:9192763392
total_system_memory:16760053760
total_system_memory_human:15.61G
used_memory_lua:39936
used_memory_lua_human:39.00K
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:1
maxmemory:13958643712
maxmemory_human:13.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.05
allocator_frag_bytes:467973376
allocator_rss_ratio:1.02
allocator_rss_bytes:175489024
rss_overhead_ratio:0.99
rss_overhead_bytes:-61399040
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:583801504
mem_not_counted_for_evict:0
mem_replication_backlog:268435456
mem_clients_slaves:46393
mem_clients_normal:52374328
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:2518313641
rdb_bgsave_in_progress:0
rdb_last_save_time:1725562546
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:34566144
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:921173960
total_commands_processed:14952159856
instantaneous_ops_per_sec:60515
total_net_input_bytes:1900864075154
total_net_output_bytes:14500081098022
instantaneous_input_kbps:5689.22
instantaneous_output_kbps:23770.42
rejected_connections:0
sync_full:2
sync_partial_ok:0
sync_partial_err:2
expired_keys:750124171
expired_stale_perc:13.81
expired_time_cap_reached_count:3171
expire_cycle_cpu_milliseconds:7834986
evicted_keys:0
keyspace_hits:12140242071
keyspace_misses:467352731
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:6862
total_forks:1
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:530043
dump_payload_sanitizations:0
total_reads_processed:10869372781
total_writes_processed:13007580135
instantaneous_lock_contention:0
avg_lock_contention:3.781250
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:2
slave0:ip=10.65.38.205,port=7000,state=online,offset=1285841055538,lag=0
slave1:ip=10.65.60.211,port=7000,state=online,offset=1285840495003,lag=0
master_failover_state:no-failover
master_replid:54033409533fc7127f630de77883bd0aceba9a52
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1285841430090
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:268435456
repl_backlog_first_byte_offset:1285572994635
repl_backlog_histlen:268435456

# CPU
used_cpu_sys:410178.050870
used_cpu_user:591817.050714
used_cpu_sys_children:0.033826
used_cpu_user_children:0.075178
server_threads:6
long_lock_waits:11777
used_cpu_sys_main_thread:53473.541689
used_cpu_user_main_thread:82091.385327

# Modules

# Commandstats
cmdstat_lrem:calls=221,usec=2890,usec_per_call=13.08,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=9923,usec=57845,usec_per_call=5.83,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=23199227,usec=40887810,usec_per_call=1.76,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=345431834,usec=10227006479,usec_per_call=29.61,rejected_calls=209443,failed_calls=0
cmdstat_hmset:calls=23226770,usec=310812343,usec_per_call=13.38,rejected_calls=119,failed_calls=0
cmdstat_config:calls=14255,usec=1018150,usec_per_call=71.42,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=2,usec=235,usec_per_call=117.50,rejected_calls=0,failed_calls=0
cmdstat_slowlog:calls=28510,usec=47868,usec_per_call=1.68,rejected_calls=0,failed_calls=0
cmdstat_set:calls=1592112444,usec=14634545551,usec_per_call=9.19,rejected_calls=222499,failed_calls=0
cmdstat_hset:calls=19590,usec=115125,usec_per_call=5.88,rejected_calls=13,failed_calls=0
cmdstat_hgetall:calls=169701464,usec=1791784949,usec_per_call=10.56,rejected_calls=0,failed_calls=0
cmdstat_replping:calls=2,usec=0,usec_per_call=0.00,rejected_calls=34,failed_calls=0
cmdstat_eval:calls=226525,usec=7508744,usec_per_call=33.15,rejected_calls=0,failed_calls=0
cmdstat_client:calls=14255,usec=30418,usec_per_call=2.13,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=994516,usec=2301629,usec_per_call=2.31,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=817899,usec=747140,usec_per_call=0.91,rejected_calls=0,failed_calls=0
cmdstat_del:calls=359369658,usec=2393904599,usec_per_call=6.66,rejected_calls=106,failed_calls=0
cmdstat_info:calls=75131,usec=4941407,usec_per_call=65.77,rejected_calls=0,failed_calls=0
cmdstat_lpush:calls=222,usec=1832,usec_per_call=8.25,rejected_calls=0,failed_calls=0
cmdstat_get:calls=12436888906,usec=28178701149,usec_per_call=2.27,rejected_calls=83574,failed_calls=0
cmdstat_latency:calls=28510,usec=98199,usec_per_call=3.44,rejected_calls=0,failed_calls=14255

# Errorstats
errorstat_CLUSTERDOWN:count=306311
errorstat_ERR:count=14255
errorstat_LOADING:count=209477

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=12666304,expires=12665000,avg_ttl=56124314,cached_keys=12666304

# KeyDB
mvcc_depth:0

and 2 hour later

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
395091:395123:M 10 Sep 2024 19:29:49.068 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
395091:395123:M 10 Sep 2024 19:29:49.068 # Accessing address: 0x544547010303
395091:395123:M 10 Sep 2024 19:29:49.068 # Crashed running the instruction at: 0x629265e56068

------ STACK TRACE ------
EIP:
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x629265e56068]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x742958642520]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x629265e56068]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](+0x1d196a) [0x629265e4296a]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](readQueryFromClient(connection*)+0x39c) [0x629265e34d0c]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x1e0) [0x629265f2a6e0]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](ProcessEventCore+0xf8) [0x629265e01338]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeProcessEvents+0x17d) [0x629265e053fd]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeMain+0x3e) [0x629265e05a6e]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](workerThreadMain(void*)+0xfe) [0x629265e1de9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x742958694ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x742958726850]

------ REGISTERS ------
395091:395123:M 10 Sep 2024 19:29:49.073 #
RAX:50fb3b8fc077d8cf RBX:0000742957ab7380
RCX:00000000e09e2d80 RDX:0000544547010303
RDI:000000000077d8cf RSI:0000000000000000
RBP:0000742673a9ab2b RSP:0000742938dfc6e0
R8 :3f8b86a80035cc71 R9 :000074269255487b
R10:0000000000000003 R11:000062926639c010
R12:000074268facd2f0 R13:0000000000060774
R14:000074269e8d4580 R15:000074268facd2f0
RIP:0000629265e56068 EFL:0000000000010206
CSGSFS:002b000000000033
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ef) -> 0000000000000000
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ee) -> 0000742900000000
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ed) -> 0000742938dfd978
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ec) -> 00000000b6b66930
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6eb) -> 000062926635bbf9
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ea) -> 000000000000b000
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e9) -> 0000742938dfc838
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e8) -> 000074250000001f
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e7) -> 0000629265e1693c
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e6) -> 0000000000000000
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e5) -> 0000000000000001
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e4) -> 00006292664dd920
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e3) -> 00007425ad438a80
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e2) -> 00007425ac7733c0
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e1) -> 00006292664dbc00
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e0) -> 0000742938dfdb38

Aditional information

  1. OS distribution and version
root@keydb2:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy
root@keydb2:~# uname -a
Linux keydb2 6.5.0-27-generic #28~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 15 10:51:06 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

We have no idea how to reproduce. Long time it was working well.

broken-ufa avatar Sep 11 '24 07:09 broken-ufa

+1 here with similar issue. After initial crash and core dump it goes into a crash loop. Restarting KeyDB helps.

365491:365502:S 11 Nov 2024 16:07:57.025 * Background saving started by pid 728172
365491:365502:S 11 Nov 2024 16:07:57.025 * Background saving started

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
728172:365502:C 11 Nov 2024 16:07:57.146 # === ASSERTION FAILED ===
728172:365502:C 11 Nov 2024 16:07:57.146 # ==> rdb.cpp:1372 'ckeysExpired == db->expireSize()' is not true

------ STACK TRACE ------

Backtrace:
keydb-rdb-bgsave *:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x4ae) [0x4f37fe]
keydb-rdb-bgsave *:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x96) [0x4f38b6]
keydb-rdb-bgsave *:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x50) [0x4f3b90]
keydb-rdb-bgsave *:6379(rdbSaveBackgroundFork(rdbSaveInfo*)+0xf8) [0x4f3d58]
keydb-rdb-bgsave *:6379(launchRdbSaveThread(unsigned long&, rdbSaveInfo*)+0x22) [0x4f3e02]
keydb-rdb-bgsave *:6379(rdbSaveBackground(rdbSaveInfo*)+0x70) [0x4f4210]
keydb-rdb-bgsave *:6379(serverCron(aeEventLoop*, long long, void*)+0x818) [0x546a88]
keydb-rdb-bgsave *:6379(aeProcessEvents+0x216) [0x54bd56]
keydb-rdb-bgsave *:6379(aeMain+0x37) [0x550c07]
keydb-rdb-bgsave *:6379(workerThreadMain(void*)+0x6d) [0x53a79d]
/lib64/libpthread.so.0(+0x81ca) [0x7fc20bf011ca]
/lib64/libc.so.6(clone+0x43) [0x7fc20bb5c8d3]

------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:92c820df436166c4
redis_mode:standalone
os:Linux 4.18.0-553.22.1.el8_10.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.5.0
process_id:728172
process_supervised:systemd
run_id:95965b0e54e1e0b71c4182e290943cf37954cb1b
tcp_port:6379
server_time_usec:1731337677013297
uptime_in_seconds:546309
uptime_in_days:6
hz:10
configured_hz:10
lru_clock:3284429
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb.conf
availability_zone:
features:cluster_mget

# Clients
connected_clients:108
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:438
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:54
thread_1_clients:55

# Memory
used_memory:20985232
used_memory_human:20.01M
used_memory_rss:29716480
used_memory_rss_human:28.34M
used_memory_peak:22945600
used_memory_peak_human:21.88M
used_memory_peak_perc:91.46%
used_memory_overhead:5510240
used_memory_startup:2091944
used_memory_dataset:15474992
used_memory_dataset_perc:81.91%
allocator_allocated:22075064
allocator_active:24076288
allocator_resident:30257152
total_system_memory:1864835072
total_system_memory_human:1.74G
used_memory_lua:44032
used_memory_lua_human:43.00K
used_memory_scripts:1016
used_memory_scripts_human:1016B
number_of_cached_scripts:5
maxmemory:1000000000
maxmemory_human:953.67M
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.09
allocator_frag_bytes:2001224
allocator_rss_ratio:1.26
allocator_rss_bytes:6180864
rss_overhead_ratio:0.98
rss_overhead_bytes:-540672
mem_fragmentation_ratio:1.41
mem_fragmentation_bytes:8651248
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:20512
mem_clients_normal:2214576
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:21382
rdb_bgsave_in_progress:0
rdb_last_save_time:1731337076
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:1404928
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:3867420
total_commands_processed:69090911
instantaneous_ops_per_sec:790
total_net_input_bytes:31497015966
total_net_output_bytes:25032556781
instantaneous_input_kbps:296.08
instantaneous_output_kbps:352.56
rejected_connections:0
sync_full:0
sync_partial_ok:1
sync_partial_err:0
expired_keys:661174
expired_stale_perc:0.59
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:49675
evicted_keys:0
keyspace_hits:8425824
keyspace_misses:7897804
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:8402
total_forks:908
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:109262
dump_payload_sanitizations:0
total_reads_processed:40746908
total_writes_processed:37551438
instantaneous_lock_contention:2
avg_lock_contention:0.265625
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:active-replica
master_global_link_status:up
connected_masters:1
master_host:192.xxx.xxx.xxx
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:33579916824
slave_repl_offset:33579916824
slave_priority:100
slave_read_only:0
replica_announced:1
connected_slaves:1
slave0:ip=192.xxx.xxx.xxx,port=6379,state=online,offset=30006960148,lag=1master_failover_state:no-failover
master_replid:31ba06b91e18f49bb1e95da834072dbcd93a52dd
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:30007038769
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:30005990194
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:0.006936
used_cpu_user:0.113402
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:2
long_lock_waits:133435
used_cpu_sys_main_thread:0.006936
used_cpu_user_main_thread:0.113402

# Modules

# Commandstats
cmdstat_rreplay:calls=35151250,usec=260598792,usec_per_call=7.41,rejected_calls=0,failed_calls=0
cmdstat_info:calls=109262,usec=26598087,usec_per_call=243.43,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=3471391,usec=204808162,usec_per_call=59.00,rejected_calls=0,failed_calls=0
cmdstat_unlink:calls=2138,usec=26912,usec_per_call=12.59,rejected_calls=0,failed_calls=0
cmdstat_get:calls=16323161,usec=85112008,usec_per_call=5.21,rejected_calls=0,failed_calls=0
cmdstat_incrby:calls=341,usec=3625,usec_per_call=10.63,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=327287,usec=581459,usec_per_call=1.78,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=3526380,usec=43214092,usec_per_call=12.25,rejected_calls=0,failed_calls=0
cmdstat_set:calls=3494310,usec=49528710,usec_per_call=14.17,rejected_calls=0,failed_calls=0
cmdstat_client:calls=109262,usec=642075,usec_per_call=5.88,rejected_calls=0,failed_calls=0
cmdstat_echo:calls=968210,usec=2667320,usec_per_call=2.75,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=69,usec=125,usec_per_call=1.81,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=40,usec=36732,usec_per_call=918.30,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=1,usec=102,usec_per_call=102.00,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=21,usec=11,usec_per_call=0.52,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=21,usec=150,usec_per_call=7.14,rejected_calls=0,failed_calls=0
cmdstat_ttl:calls=398,usec=1520,usec_per_call=3.82,rejected_calls=0,failed_calls=0
cmdstat_replping:calls=1,usec=2,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_decrby:calls=57,usec=293,usec_per_call=5.14,rejected_calls=0,failed_calls=0
cmdstat_latency:calls=218524,usec=2212858,usec_per_call=10.13,rejected_calls=0,failed_calls=109262
cmdstat_replconf:calls=541459,usec=1337313,usec_per_call=2.47,rejected_calls=0,failed_calls=0
cmdstat_config:calls=109262,usec=19194593,usec_per_call=175.67,rejected_calls=0,failed_calls=0
cmdstat_slowlog:calls=218524,usec=2480527,usec_per_call=11.35,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=80056,usec=543490,usec_per_call=6.79,rejected_calls=0,failed_calls=0
cmdstat_select:calls=966612,usec=1654509,usec_per_call=1.71,rejected_calls=0,failed_calls=0
cmdstat_del:calls=3472874,usec=13991739,usec_per_call=4.03,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_ERR:count=109262

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=2101,expires=2103,avg_ttl=6849142,cached_keys=2101

# KeyDB
mvcc_depth:0

# Clients...

.O.O.O.O.O.O.O.728172:365502:C 11 Nov 2024 16:07:57.471 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
728172:365502:C 11 Nov 2024 16:07:57.471 # Accessing address: 0x3c7cff8
728172:365502:C 11 Nov 2024 16:07:57.471 # Crashed running the instruction at: 0x7fc203c77d2f

Crash loop:

Nov 12 16:07:43 redis1a systemd-coredump[1612895]: Process 1612893 (keydb-server) of user 993 dumped core.
Nov 12 16:07:44 redis1a systemd-coredump[1612904]: Process 1612902 (keydb-server) of user 993 dumped core.
Nov 12 16:07:45 redis1a systemd-coredump[1612914]: Process 1612912 (keydb-server) of user 993 dumped core.
Nov 12 16:07:46 redis1a systemd-coredump[1612923]: Process 1612921 (keydb-server) of user 993 dumped core.
Nov 12 16:07:47 redis1a systemd-coredump[1612933]: Process 1612931 (keydb-server) of user 993 dumped core.
Nov 12 16:07:48 redis1a systemd-coredump[1612942]: Process 1612940 (keydb-server) of user 993 dumped core.
Nov 12 16:07:49 redis1a systemd-coredump[1612952]: Process 1612949 (keydb-server) of user 993 dumped core.

OS info:

AlmaLinux release 8.10 (Cerulean Leopard)
Linux redis1a 4.18.0-553.22.1.el8_10.x86_64 #1 SMP Tue Sep 24 05:16:59 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

matejzero avatar Nov 12 '24 15:11 matejzero

Hi @matejzero

Your crash is different from the one reported by @broken-ufa. You may want to have a read of this issue: https://github.com/Snapchat/KeyDB/issues/743

keithchew avatar Nov 13 '24 12:11 keithchew

Hi @broken-ufa

I found 2 crashes when m_pdict is accessed, but in dictRehash() instead of your prefetchKeysAsync method (my instance does not go through that code flow because I have Flash enabled):

https://github.com/Snapchat/KeyDB/issues/792 https://github.com/Snapchat/KeyDB/issues/876

I have not found the cause of the crashes, but I have disabled activereshash to workaround them, ie:

activerehashing no

Maybe you an also try this config and see if you still get your crash.

keithchew avatar Nov 13 '24 13:11 keithchew

Thank you! I will try it I will come back with updates

broken-ufa avatar Nov 15 '24 16:11 broken-ufa