KeyDB

KeyDB copied to clipboard

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

1:24:M 02 Jan 2023 04:42:24.096 * 10 changes in 300 seconds. Saving...
1:24:M 02 Jan 2023 04:42:24.096 * Background saving started
1:46:M 02 Jan 2023 04:42:24.103 * DB saved on disk
1:46:M 02 Jan 2023 04:42:24.105 * RDB: 21 MB of memory used by copy-on-write
1:24:M 02 Jan 2023 04:42:24.197 * Background saving terminated with success


=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:24:M 02 Jan 2023 04:46:40.240 # ------------------------------------------------
1:24:M 02 Jan 2023 04:46:40.240 # !!! Software Failure. Press left mouse button to continue
1:24:M 02 Jan 2023 04:46:40.240 # Guru Meditation: Unknown btype in replyToBlockedClientTimedOut(). #blocked.cpp:246

------ STACK TRACE ------

Backtrace:
keydb-server *:6379(replyToBlockedClientTimedOut(client*)+0x72) [0x5591b9872552]
keydb-server *:6379(checkBlockedClientTimeout(client*, long long)+0x49) [0x5591b992eb99]
keydb-server *:6379(handleBlockedClientsTimeout()+0x127) [0x5591b992ecd7]
keydb-server *:6379(beforeSleep(aeEventLoop*)+0x155) [0x5591b9866d35]
keydb-server *:6379(aeProcessEvents+0x10b) [0x5591b985ea1b]
keydb-server *:6379(aeMain+0x47) [0x5591b9868467]
keydb-server *:6379(workerThreadMain(void*)+0x73) [0x5591b99384b3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fdc792ac6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fdc78fd561f]

------ INFO OUTPUT ------
# Server
redis_version:6.3.1
redis_git_sha1:ee16abf0
redis_git_dirty:1
redis_build_id:46da54e1e1000581
redis_mode:standalone
os:Linux 6.0.9-arch1-1 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:1
process_supervised:no
run_id:f0c8c2f6c105d5520f28be372d186b81d579f53c
tcp_port:6379
server_time_usec:1672634800242286
uptime_in_seconds:139008
uptime_in_days:1
hz:10
configured_hz:10
lru_clock:11690416
executable:/data/keydb-server
config_file:/etc/keydb/keydb.conf

# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:1
current_client_thread:0
thread_0_clients:2
thread_1_clients:0

# Memory
used_memory:2119752
used_memory_human:2.02M
used_memory_rss:27824128
used_memory_rss_human:26.54M
used_memory_peak:278921568
used_memory_peak_human:266.00M
used_memory_peak_perc:0.76%
used_memory_overhead:2038824
used_memory_startup:1989496
used_memory_dataset:80928
used_memory_dataset_perc:62.13%
allocator_allocated:2750672
allocator_active:5664768
allocator_resident:13594624
total_system_memory:33577005056
total_system_memory_human:31.27G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:2.06
allocator_frag_bytes:2914096
allocator_rss_ratio:2.40
allocator_rss_bytes:7929856
rss_overhead_ratio:2.05
rss_overhead_bytes:14229504
mem_fragmentation_ratio:13.62
mem_fragmentation_bytes:25780704
mem_not_counted_for_evict:1048576
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:41008
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none

# Persistence
loading:0
current_cow_size:22478848
current_cow_size_age:256
current_fork_perc:0.00
current_save_keys_processed:1
current_save_keys_total:0
rdb_changes_since_last_save:44
rdb_bgsave_in_progress:0
rdb_last_save_time:1672634544
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:22478848
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:40
total_commands_processed:124635
instantaneous_ops_per_sec:0
total_net_input_bytes:134988398
total_net_output_bytes:975706
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:1026
keyspace_misses:10
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:21
dump_payload_sanitizations:0
total_reads_processed:81319
total_writes_processed:80457
instantaneous_lock_contention:2
avg_lock_contention:0.000000
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:c20878f308c580b64eb78ad342b06d5256e7eb0c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:162.387607
used_cpu_user:366.253957
used_cpu_sys_children:0.001478
used_cpu_user_children:0.001538
server_threads:2
long_lock_waits:8
used_cpu_sys_main_thread:94.482873
used_cpu_user_main_thread:210.632029

# Modules

# Commandstats
cmdstat_unlink:calls=30378,usec=28271,usec_per_call=0.93,rejected_calls=0,failed_calls=0
cmdstat_command:calls=1,usec=839,usec_per_call=839.00,rejected_calls=0,failed_calls=0
cmdstat_blpop:calls=25,usec=299,usec_per_call=11.96,rejected_calls=0,failed_calls=0
cmdstat_hmset:calls=92643,usec=2952514,usec_per_call=31.87,rejected_calls=0,failed_calls=0
cmdstat_config:calls=30,usec=335,usec_per_call=11.17,rejected_calls=0,failed_calls=0
cmdstat_rpush:calls=334,usec=2869,usec_per_call=8.59,rejected_calls=0,failed_calls=0
cmdstat_flushdb:calls=1,usec=131936,usec_per_call=131936.00,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=21,usec=332,usec_per_call=15.81,rejected_calls=0,failed_calls=0
cmdstat_client:calls=20,usec=48,usec_per_call=2.40,rejected_calls=0,failed_calls=0
cmdstat_select:calls=19,usec=28,usec_per_call=1.47,rejected_calls=0,failed_calls=0
cmdstat_info:calls=76,usec=6130,usec_per_call=80.66,rejected_calls=0,failed_calls=0
cmdstat_hdel:calls=3,usec=4,usec_per_call=1.33,rejected_calls=0,failed_calls=0
cmdstat_get:calls=10,usec=27,usec_per_call=2.70,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=16,usec=112,usec_per_call=7.00,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=10,usec=7,usec_per_call=0.70,rejected_calls=0,failed_calls=10
cmdstat_echo:calls=20,usec=8,usec_per_call=0.40,rejected_calls=0,failed_calls=0
cmdstat_subscribe:calls=10,usec=33,usec_per_call=3.30,rejected_calls=0,failed_calls=0
cmdstat_del:calls=12,usec=25,usec_per_call=2.08,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=1005,usec=1218,usec_per_call=1.21,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=1,usec=30,usec_per_call=30.00,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_ERR:count=21

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=3,expires=0,avg_ttl=0,cached_keys=3

# KeyDB
mvcc_depth:1

------ CLIENT LIST OUTPUT ------
id=17 addr=127.0.0.1:35486 laddr=127.0.0.1:6379 fd=20 name= age=571 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=6 obl=0 oll=0 omem=0 tot-mem=61462 events=r cmd=keys user=default redir=-1
id=43 addr=172.17.0.1:52866 laddr=172.17.0.5:6379 fd=31 name= age=13 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
1:24:M 02 Jan 2023 04:46:40.242 # main thread terminated
1:24:M 02 Jan 2023 04:46:40.242 # Bio thread for job type #0 terminated
1:24:M 02 Jan 2023 04:46:40.242 # Bio thread for job type #1 terminated
1:24:M 02 Jan 2023 04:46:40.242 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== KEYDB BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/JohnSully/KeyDB/issues

  Suspect RAM error? Use keydb-server --test-memory to verify it.

Aditional information

1. OS distribution and version Docker eqalpha/keydb a7d31975720e, on Arch Linux host

2. Steps to reproduce (if any) I don't know how to reproduce this.

Even if you are not sure how to reproduce this, context on the configs you are using and the commands that were running can help us investigate.

Sure, no problem. I was running the default configuration: docker run --name redis -p localhost:6379:6379 eqalpha/keydb, and was doing HSET commands, ~4k/sec.

Hi there, I have also encountered this crash. I was running KeyDB as a systemd service.

Logs, config file and systemd unit file are attached: keydb-debugging.zip