KeyDB
KeyDB copied to clipboard
Published
20 hours ago •
Snapchat
Snapchat
[CRASH] Slave node crashes with an assertion failure at: networking.cpp:2753'FCorrectThread(c)' is not true
Crash report
In a cache only cluster setup with 3 master/ 3 replicas, when we run a long stability testing (for couple of days) using a script which uses memtier benchmark to write / read large data continuously, we observed a slave crash with following stack race.
...
1:20:S 16 May 2023 18:25:40.198 * Full resync from master: 51b08be11f33bc3477d8e838bb214710c33f74f0:1406119098112
1:20:S 16 May 2023 18:25:40.198 * Discarding previously cached master state.
1:20:S 16 May 2023 18:25:51.884 * MASTER <-> REPLICA sync: receiving 2670464322 bytes from master to disk
1:20:S 16 May 2023 18:26:28.677 # I/O error trying to sync with MASTER: connection lost
1:20:S 16 May 2023 18:26:28.684 * Reconnecting to MASTER 172.31.11.1:6379 after failure
1:20:S 16 May 2023 18:26:28.687 * MASTER <-> REPLICA sync started
1:20:S 16 May 2023 18:26:28.687 * Non blocking connect for SYNC fired the event.
1:20:S 16 May 2023 18:26:28.691 * Master replied to PING, replication can continue...
1:20:S 16 May 2023 18:26:28.697 * Partial resynchronization not possible (no cached master)
1:20:S 16 May 2023 18:26:28.699 * Full resync from master: 51b08be11f33bc3477d8e838bb214710c33f74f0:1407194840336
1:20:S 16 May 2023 18:26:40.551 * MASTER <-> REPLICA sync: receiving 2670122844 bytes from master to disk
1:20:S 16 May 2023 18:27:00.378 * Killing running AOF rewrite child: 70
70:signal-handler (1684261620) Received SIGUSR1 in child, exiting now.
1:20:S 16 May 2023 18:27:00.826 * MASTER <-> REPLICA sync: Flushing old data
1:20:S 16 May 2023 18:27:00.830 * MASTER <-> REPLICA sync: Loading DB in memory
1:20:S 16 May 2023 18:27:00.839 * Loading RDB produced by version 255.255.255
1:20:S 16 May 2023 18:27:00.840 * RDB age 32 seconds
1:20:S 16 May 2023 18:27:00.840 * RDB memory usage when created 4328.19 Mb
1:20:S 16 May 2023 18:27:25.688 # Done loading RDB, keys loaded: 0, keys expired: 0.
1:20:S 16 May 2023 18:39:58.844 * Full resync from master: 51b08be11f33bc3477d8e838bb214710c33f74f0:1427138362160
1:20:S 16 May 2023 18:39:58.844 * Discarding previously cached master state.
1:20:S 16 May 2023 18:40:12.300 * MASTER <-> REPLICA sync: receiving 2672533755 bytes from master to disk
1:20:S 16 May 2023 18:40:29.008 # I/O error trying to sync with MASTER: connection lost
1:20:S 16 May 2023 18:40:29.014 * Reconnecting to MASTER 172.31.11.1:6379 after failure
1:20:S 16 May 2023 18:40:29.015 * MASTER <-> REPLICA sync started
1:20:S 16 May 2023 18:40:29.015 * Non blocking connect for SYNC fired the event.
1:20:S 16 May 2023 18:40:29.027 * Master replied to PING, replication can continue...
1:20:S 16 May 2023 18:40:29.037 * Partial resynchronization not possible (no cached master)
1:20:S 16 May 2023 18:40:29.040 * Full resync from master: 51b08be11f33bc3477d8e838bb214710c33f74f0:1428215484462
1:20:S 16 May 2023 18:40:41.176 * MASTER <-> REPLICA sync: receiving 2670698896 bytes from master to disk
1:20:S 16 May 2023 18:41:12.528 * Killing running AOF rewrite child: 79
79:signal-handler (1684262472) Received SIGUSR1 in child, exiting now.
1:20:S 16 May 2023 18:41:12.964 * MASTER <-> REPLICA sync: Flushing old data
1:20:S 16 May 2023 18:41:12.967 * MASTER <-> REPLICA sync: Loading DB in memory
1:20:S 16 May 2023 18:41:12.975 * Loading RDB produced by version 255.255.255
1:20:S 16 May 2023 18:41:12.975 * RDB age 43 seconds
1:20:S 16 May 2023 18:41:12.976 * RDB memory usage when created 4328.30 Mb
1:17:S 16 May 2023 18:42:13.755 # Timeout receiving bulk data from MASTER... If the problem persists try to set the 'repl-timeout' parameter in keydb.conf to a larger value.
1:17:S 16 May 2023 18:42:13.762 * Reconnecting to MASTER 172.31.11.1:6379 after failure
1:17:S 16 May 2023 18:42:13.763 * MASTER <-> REPLICA sync started
1:17:S 16 May 2023 18:42:13.764 * Non blocking connect for SYNC fired the event.
1:17:S 16 May 2023 18:42:13.826 * Master replied to PING, replication can continue...
1:17:S 16 May 2023 18:42:13.985 * Partial resynchronization not possible (no cached master)
1:17:S 16 May 2023 18:42:14.049 * Full resync from master: 51b08be11f33bc3477d8e838bb214710c33f74f0:1430489052045
1:20:S 16 May 2023 18:42:26.052 # Done loading RDB, keys loaded: 0, keys expired: 0.
1:20:S 16 May 2023 18:42:26.054 * MASTER <-> REPLICA sync: Finished with success
1:20:S 16 May 2023 18:42:26.408 * Background append only file rewriting started by pid 81
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:17:S 16 May 2023 18:42:26.495 # === ASSERTION FAILED ===
1:20:S 16 May 2023 18:42:26.495 # WARNING: Receiving inline protocol from master, master stream corruption? Closing the master connection and discarding the cached master.
1:17:S 16 May 2023 18:42:26.495 # ==> networking.cpp:2753 'FCorrectThread(c)' is not true
1:20:S 16 May 2023 18:42:26.496 # Protocol error (Master using the inline protocol. Desync?) from client: id=164965 addr=172.31.11.1:6379 laddr=172.31.26.148:29355 fd=39 name= age=0 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=13 qbuf-free=40941 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61440 events=r cmd=NULL user=(superuser) redir=-1 peer=172.31.11.1:6379 network=vpc. Query buffer during protocol error: '$2671703771..'
------ STACK TRACE ------
Backtrace:
/opt/redis/redis-server 0.0.0.0:6379 [cluster](readQueryFromClient(connection*)+0x710) [0x611190]
/opt/redis/redis-server 0.0.0.0:6379 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x11e) [0x66b67e]
/opt/redis/redis-server 0.0.0.0:6379 [cluster](ProcessEventCore+0xfc) [0x598eec]
/opt/redis/redis-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x205) [0x59a565]
/opt/redis/redis-server 0.0.0.0:6379 [cluster](aeMain+0x37) [0x5a1f47]
/opt/redis/redis-server 0.0.0.0:6379 [cluster](workerThreadMain(void*)+0x64) [0x67e8f4]
/usr/lib64/libpthread.so.0(+0x8f4b) [0x7f3489ec4f4b]
/usr/lib64/libc.so.6(clone+0x3f) [0x7f3489dfc7ef]
------ INFO OUTPUT ------
# Server
redis_version:255.255.255
patch_version:6.2.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c0eb0c37c47263fe
redis_mode:cluster
os:Linux
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:0.0.0
process_id:1
process_supervised:no
run_id:c8537429d6244fa2c30ffe5bdfcac2b26d31c447
tcp_port:6379
server_time_usec:1684262546527084
uptime_in_seconds:119151
uptime_in_days:1
hz:10
configured_hz:10
lru_clock:6540946
executable:redis-server
config_file:redis.conf
instance_id:371a04f3-6328-431c-a6c3-fbcdeb74c250
# Clients
connected_clients:85
cluster_connections:10
maxclients:10010
client_recent_max_input_buffer:24
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
rx_controlled_clients:0
total_rx_controlled:0
proxy_header_error:0
thread_0_clients:26
thread_1_clients:21
thread_2_clients:24
thread_3_clients:14
# Memory
used_memory:14656079344
used_memory_human:13.65G
used_memory_rss:7210586112
used_memory_rss_human:6.72G
used_memory_peak:14779592728
used_memory_peak_human:13.76G
used_memory_peak_perc:99.16%
used_memory_overhead:1313258936
used_memory_startup:4678760
used_memory_dataset:13342820408
used_memory_dataset_perc:91.07%
allocator_allocated:14758036376
allocator_active:14888771584
allocator_resident:15155089408
total_system_memory:8174989312
total_system_memory_human:7.61G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:8589934592
maxmemory_human:8.00G
maxmemory_policy:volatile-lru
allocator_frag_ratio:1.01
allocator_frag_bytes:130735208
allocator_rss_ratio:1.02
allocator_rss_bytes:266317824
rss_overhead_ratio:0.48
rss_overhead_bytes:-7944503296
mem_fragmentation_ratio:0.49
mem_fragmentation_bytes:-7546128688
mem_not_counted_for_evict:1073741828
mem_replication_backlog:1073741824
mem_clients_slaves:0
mem_clients_normal:1250744
mem_aof_buffer:8
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:30004974
lazyfreed_objects:30004974
storage_provider:none
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:5000829
rdb_changes_since_last_save:2944861300
rdb_bgsave_in_progress:0
rdb_last_save_time:1684143395
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:1
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:29
aof_current_rewrite_time_sec:0
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:2288291840
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:16900733165
aof_base_size:3242364952
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:164952
total_commands_processed:2945182203
instantaneous_ops_per_sec:0
total_net_input_bytes:1437813791225
total_net_output_bytes:252650950
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:348175
total_forks:53
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:2834
dump_payload_sanitizations:0
total_reads_processed:286773164
total_writes_processed:301439
instantaneous_lock_contention:0
avg_lock_contention:3.343750
storage_provider_read_hits:0
storage_provider_read_misses:0
# Replication
role:slave
master_global_link_status:up
master_host:172.31.11.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:1430489068442
slave_repl_offset:1430489052045
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:51b08be11f33bc3477d8e838bb214710c33f74f0
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1430489052045
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1073741824
repl_backlog_first_byte_offset:1430489052046
repl_backlog_histlen:0
# CPU
used_cpu_sys:6117.884450
used_cpu_user:12091.252318
used_cpu_sys_children:160.917791
used_cpu_user_children:391.676093
server_threads:4
long_lock_waits:72233
used_cpu_sys_main_thread:118.678263
used_cpu_user_main_thread:565.923592
# Modules
module:name=announce,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=traffic_control,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=auth,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_dcs.updateauth:calls=1,usec=104,usec_per_call=104.00,rejected_calls=0,failed_calls=0
cmdstat_bgrewriteaof:calls=46,usec=4135379,usec_per_call=89899.55,rejected_calls=0,failed_calls=0
cmdstat_select:calls=5,usec=15,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_acl:calls=2,usec=71,usec_per_call=35.50,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=146990,usec=2216770,usec_per_call=15.08,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=128314,usec=3195156,usec_per_call=24.90,rejected_calls=2834,failed_calls=0
cmdstat_info:calls=28802,usec=2933799,usec_per_call=101.86,rejected_calls=0,failed_calls=0
cmdstat_slowlog:calls=3972,usec=61850,usec_per_call=15.57,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=11773,usec=17390,usec_per_call=1.48,rejected_calls=0,failed_calls=0
cmdstat_config:calls=996,usec=32065,usec_per_call=32.19,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=2,usec=2,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_set:calls=2944861300,usec=5237824955,usec_per_call=1.78,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_LOADING:count=2834
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=5000829,expires=0,avg_ttl=0,cached_keys=5000829
# KeyDB
variant:enterprise
license_status:OK
mvcc_depth:0
------ CLIENT LIST OUTPUT ------
id=164882 addr=192.168.160.199:57926 laddr=192.168.174.224:6379 fd=78 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:57926 network=vpc
id=164883 addr=192.168.160.199:57994 laddr=192.168.174.224:6379 fd=74 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:57994 network=vpc
id=164884 addr=192.168.160.199:58064 laddr=192.168.174.224:6379 fd=52 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58064 network=vpc
id=164885 addr=192.168.160.199:57966 laddr=192.168.174.224:6379 fd=45 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:57966 network=vpc
id=164886 addr=192.168.160.199:57946 laddr=192.168.174.224:6379 fd=86 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:57946 network=vpc
id=164887 addr=192.168.160.199:58208 laddr=192.168.174.224:6379 fd=90 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58208 network=vpc
id=164888 addr=192.168.160.199:58162 laddr=192.168.174.224:6379 fd=93 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58162 network=vpc
id=164904 addr=192.168.160.199:58402 laddr=192.168.174.224:6379 fd=41 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58402 network=vpc
id=164905 addr=192.168.160.199:58426 laddr=192.168.174.224:6379 fd=40 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58426 network=vpc
id=164906 addr=192.168.160.199:58450 laddr=192.168.174.224:6379 fd=42 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58450 network=vpc
id=164907 addr=192.168.160.199:58476 laddr=192.168.174.224:6379 fd=44 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58476 network=vpc
id=164908 addr=192.168.160.199:58394 laddr=192.168.174.224:6379 fd=43 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58394 network=vpc
id=164909 addr=192.168.160.199:58498 laddr=192.168.174.224:6379 fd=46 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58498 network=vpc
id=164910 addr=192.168.160.199:58462 laddr=192.168.174.224:6379 fd=47 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58462 network=vpc
id=164911 addr=192.168.160.199:58526 laddr=192.168.174.224:6379 fd=48 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58526 network=vpc
id=164912 addr=192.168.160.199:58486 laddr=192.168.174.224:6379 fd=49 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58486 network=vpc
id=164913 addr=192.168.160.199:58536 laddr=192.168.174.224:6379 fd=50 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58536 network=vpc
id=164914 addr=192.168.160.199:58514 laddr=192.168.174.224:6379 fd=51 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58514 network=vpc
id=164915 addr=192.168.160.199:58546 laddr=192.168.174.224:6379 fd=53 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58546 network=vpc
id=164916 addr=192.168.160.199:58414 laddr=192.168.174.224:6379 fd=54 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58414 network=vpc
id=164917 addr=192.168.160.199:58570 laddr=192.168.174.224:6379 fd=55 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58570 network=vpc
id=164918 addr=192.168.160.199:58584 laddr=192.168.174.224:6379 fd=56 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58584 network=vpc
id=164919 addr=192.168.160.199:58440 laddr=192.168.174.224:6379 fd=57 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58440 network=vpc
id=164966 addr=/opt/redis/redis.sock:0 laddr=/opt/redis/redis.sock:0 fd=124 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=info user=default redir=-1 peer=/opt/redis/redis.sock:0 network=bms
id=164920 addr=192.168.160.199:58622 laddr=192.168.174.224:6379 fd=58 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58622 network=vpc
id=164921 addr=192.168.160.199:58668 laddr=192.168.174.224:6379 fd=59 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58668 network=vpc
id=164922 addr=192.168.160.199:58562 laddr=192.168.174.224:6379 fd=60 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58562 network=vpc
id=164923 addr=192.168.160.199:58654 laddr=192.168.174.224:6379 fd=61 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58654 network=vpc
id=164924 addr=192.168.160.199:58706 laddr=192.168.174.224:6379 fd=62 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58706 network=vpc
id=164925 addr=192.168.160.199:58596 laddr=192.168.174.224:6379 fd=63 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58596 network=vpc
id=164926 addr=192.168.160.199:58630 laddr=192.168.174.224:6379 fd=64 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58630 network=vpc
id=164927 addr=192.168.160.199:58728 laddr=192.168.174.224:6379 fd=65 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58728 network=vpc
id=164928 addr=192.168.160.199:58738 laddr=192.168.174.224:6379 fd=66 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58738 network=vpc
id=164929 addr=192.168.160.199:58606 laddr=192.168.174.224:6379 fd=67 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58606 network=vpc
id=164930 addr=192.168.160.199:58682 laddr=192.168.174.224:6379 fd=68 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58682 network=vpc
id=164931 addr=192.168.160.199:58824 laddr=192.168.174.224:6379 fd=69 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58824 network=vpc
id=164932 addr=192.168.160.199:58762 laddr=192.168.174.224:6379 fd=70 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58762 network=vpc
id=164933 addr=192.168.160.199:58642 laddr=192.168.174.224:6379 fd=71 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58642 network=vpc
id=164934 addr=192.168.160.199:58716 laddr=192.168.174.224:6379 fd=72 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58716 network=vpc
id=164935 addr=192.168.160.199:58834 laddr=192.168.174.224:6379 fd=73 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58834 network=vpc
id=164936 addr=192.168.160.199:58774 laddr=192.168.174.224:6379 fd=75 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58774 network=vpc
id=164937 addr=192.168.160.199:58692 laddr=192.168.174.224:6379 fd=76 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58692 network=vpc
id=164938 addr=192.168.160.199:58750 laddr=192.168.174.224:6379 fd=77 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58750 network=vpc
id=164939 addr=192.168.160.199:58982 laddr=192.168.174.224:6379 fd=79 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58982 network=vpc
id=164940 addr=192.168.160.199:58850 laddr=192.168.174.224:6379 fd=80 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58850 network=vpc
id=164941 addr=192.168.160.199:58786 laddr=192.168.174.224:6379 fd=81 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58786 network=vpc
id=164942 addr=192.168.160.199:58798 laddr=192.168.174.224:6379 fd=82 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58798 network=vpc
id=164943 addr=192.168.160.199:59002 laddr=192.168.174.224:6379 fd=83 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59002 network=vpc
id=164944 addr=192.168.160.199:58872 laddr=192.168.174.224:6379 fd=84 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58872 network=vpc
id=164945 addr=192.168.160.199:58812 laddr=192.168.174.224:6379 fd=85 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58812 network=vpc
id=164946 addr=192.168.160.199:58860 laddr=192.168.174.224:6379 fd=87 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58860 network=vpc
id=164947 addr=192.168.160.199:59018 laddr=192.168.174.224:6379 fd=88 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59018 network=vpc
id=164948 addr=192.168.160.199:58882 laddr=192.168.174.224:6379 fd=89 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58882 network=vpc
id=164949 addr=192.168.160.199:58918 laddr=192.168.174.224:6379 fd=91 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58918 network=vpc
id=164950 addr=192.168.160.199:59052 laddr=192.168.174.224:6379 fd=92 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59052 network=vpc
id=164965 addr=172.31.11.1:6379 laddr=172.31.26.148:29355 fd=39 name= age=0 idle=0 flags=Mc db=0 sub=0 psub=0 multi=-1 qbuf=16397 qbuf-free=24557 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61440 events=r cmd=NULL user=(superuser) redir=-1 peer=172.31.11.1:6379 network=vpc
id=164951 addr=192.168.160.199:59026 laddr=192.168.174.224:6379 fd=94 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59026 network=vpc
id=164952 addr=192.168.160.199:58896 laddr=192.168.174.224:6379 fd=95 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58896 network=vpc
id=164953 addr=192.168.160.199:58946 laddr=192.168.174.224:6379 fd=97 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58946 network=vpc
id=164954 addr=192.168.160.199:59040 laddr=192.168.174.224:6379 fd=98 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59040 network=vpc
id=164955 addr=192.168.160.199:58908 laddr=192.168.174.224:6379 fd=100 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58908 network=vpc
id=164956 addr=192.168.160.199:58970 laddr=192.168.174.224:6379 fd=101 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58970 network=vpc
id=164957 addr=192.168.160.199:59110 laddr=192.168.174.224:6379 fd=103 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59110 network=vpc
id=164958 addr=192.168.160.199:58932 laddr=192.168.174.224:6379 fd=104 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58932 network=vpc
id=164959 addr=192.168.160.199:58990 laddr=192.168.174.224:6379 fd=105 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58990 network=vpc
id=164960 addr=192.168.160.199:58958 laddr=192.168.174.224:6379 fd=106 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58958 network=vpc
id=164961 addr=192.168.160.199:59066 laddr=192.168.174.224:6379 fd=107 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59066 network=vpc
id=164962 addr=192.168.160.199:59090 laddr=192.168.174.224:6379 fd=108 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59090 network=vpc
id=164963 addr=192.168.160.199:59078 laddr=192.168.174.224:6379 fd=110 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59078 network=vpc
id=164964 addr=192.168.160.199:59098 laddr=192.168.174.224:6379 fd=111 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:59098 network=vpc
id=164889 addr=192.168.160.199:58232 laddr=192.168.174.224:6379 fd=96 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58232 network=vpc
id=164890 addr=192.168.160.199:58018 laddr=192.168.174.224:6379 fd=99 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58018 network=vpc
id=164891 addr=192.168.160.199:58254 laddr=192.168.174.224:6379 fd=102 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58254 network=vpc
id=164892 addr=192.168.160.199:58186 laddr=192.168.174.224:6379 fd=109 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58186 network=vpc
id=164893 addr=192.168.160.199:58040 laddr=192.168.174.224:6379 fd=114 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58040 network=vpc
id=164894 addr=192.168.160.199:58330 laddr=192.168.174.224:6379 fd=115 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58330 network=vpc
id=164895 addr=192.168.160.199:58280 laddr=192.168.174.224:6379 fd=118 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58280 network=vpc
id=164896 addr=192.168.160.199:58090 laddr=192.168.174.224:6379 fd=120 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58090 network=vpc
id=164897 addr=192.168.160.199:58302 laddr=192.168.174.224:6379 fd=123 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58302 network=vpc
id=164898 addr=192.168.160.199:58112 laddr=192.168.174.224:6379 fd=125 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58112 network=vpc
id=164899 addr=192.168.160.199:58346 laddr=192.168.174.224:6379 fd=126 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58346 network=vpc
id=164900 addr=192.168.160.199:58138 laddr=192.168.174.224:6379 fd=127 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58138 network=vpc
id=164901 addr=192.168.160.199:58354 laddr=192.168.174.224:6379 fd=128 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58354 network=vpc
id=164902 addr=192.168.160.199:58382 laddr=192.168.174.224:6379 fd=130 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58382 network=vpc
id=164903 addr=192.168.160.199:58368 laddr=192.168.174.224:6379 fd=131 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1 peer=192.168.160.199:58368 network=vpc
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
1:17:S 16 May 2023 18:42:26.546 # main thread terminated
1:17:S 16 May 2023 18:42:26.547 # Bio thread for job type #0 terminated
1:17:S 16 May 2023 18:42:26.548 # Bio thread for job type #1 terminated
1:17:S 16 May 2023 18:42:26.570 # Bio thread for job type #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== KEYDB BUG REPORT END. Make sure to include from START to END. ===
Aditional information
-
OS distribution and version
-
Steps to reproduce (if any): It may not be easy to reproduce the issue, however we used following script to run long stability test for several days:
for ((i=-2000000000;i<2000000000;i++)) do echo "$i th set" ./memtier_benchmark -s xxxxx -p 6379 -t 10 -c 50 -n 30000 --ratio=1:0 --random-data --data-size-range=1-1024 --data-size-pattern=S --key-minimum=1 --key-maximum=15000000 --key-pattern=P:P --cluster-mode echo "get $i" ./memtier_benchmark -s xxxxx -p 6379 -t 10 -c 50 -n 30000 --ratio=0:1 --random-data --data-size-range=1-1024 --data-size-pattern=S --key-minimum=1 --key-maximum=15000000 --key-pattern=P:P --cluster-mode doneAn example master conf:
bind 0.0.0.0 port 6379 protected-mode no dir "/opt/redis" dbfilename "redis.rdb" pidfile "/opt/redis/redis.pid" logfile "/var/log/redis.log" notify-keyspace-events "xE" tcp-keepalive 30 timeout 0 maxmemory 8gb maxclients 10010 save "" unixsocket "/opt/redis/redis.sock" unixsocketperm 600 client-output-buffer-limit replica 286331153 286331153 60 tcp-backlog 10000 lazyfree-lazy-eviction yes lazyfree-lazy-expire yes lazyfree-lazy-server-del yes replica-lazy-flush yes rdbcompression no databases 1 appendonly yes appendfilename "appendonly.aof" appendfsync no no-appendfsync-on-rewrite yes auto-aof-rewrite-percentage 0 cluster-enabled yes cluster-config-file "node.conf" cluster-node-timeout 15000 cluster-replica-validity-factor 0 cluster-require-full-coverage no cluster-migration-barrier 9999 lazyfree-lazy-user-flush yes maxmemory-policy volatile-lru hash-max-ziplist-entries 512 hash-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 latency-monitor-threshold 0 repl-backlog-size 1gb repl-backlog-ttl 3600 slowlog-log-slower-than 10000 slowlog-max-len 128 lua-time-limit 5000 repl-timeout 60 proto-max-bulk-len 512mb master-read-only no server-threads 4 min-clients-per-thread 50 server-thread-affinity false enable-async-commands yes always-show-logo yes
@JohnSully / @msotheeswaran Appreciate it, if you could help identify the possible root cause for the above problem. thanks.
@paulmchen @hengku @yzhao244
