valkey
valkey copied to clipboard
valkey-io
[CRASH] Crash in Valkey 8.0.3 during downgrade from Valkey 8.1.0
Crash report
2025-07-08T13:28:57.327427789Z I0708 15:28:57.327255 1 *:344]
2025-07-08T13:28:58.458834692Z I0708 15:28:58.458719 1 *:174] 19:S Connecting to PRIMARY 192.168.26.185:6380
2025-07-08T13:28:58.458844439Z I0708 15:28:58.458746 1 *:174] 19:S PRIMARY <-> REPLICA sync started
2025-07-08T13:28:58.458847810Z I0708 15:28:58.458797 1 *:174] 19:S Node 07e99ff38fdee190a2f0b4a5178206b408da8765 () is no longer primary of shard 534c3a48ebc3ef9cef63a12bc23c057708e2668e; removed all 0 slot(s) it used to own
2025-07-08T13:28:58.458850597Z I0708 15:28:58.458804 1 *:174] 19:S Node 07e99ff38fdee190a2f0b4a5178206b408da8765 () is now part of shard 7d1495a9919f372c9217bef2bd1ce29bc0ad120b
2025-07-08T13:28:58.458853227Z I0708 15:28:58.458809 1 *:174] 19:S Node 07e99ff38fdee190a2f0b4a5178206b408da8765 () is now a replica of node 5f63c9d2eebe3febbda7fee6cee02b1677761ade () in shard 7d1495a9919f372c9217bef2bd1ce29bc0ad120b
2025-07-08T13:28:58.458973937Z I0708 15:28:58.458920 1 *:174] 19:S Shard id bb579abc9e4a814cf4f719203a1eefa38b88e65c update request for node id 07e99ff38fdee190a2f0b4a5178206b408da8765 diverges from existing primary shard id 7d1495a9919f372c9217bef2bd1ce29bc0ad120b, rejecting!
2025-07-08T13:28:58.462079264Z I0708 15:28:58.461986 1 *:174] 19:S Non blocking connect for SYNC fired the event.
2025-07-08T13:28:58.464315290Z I0708 15:28:58.464190 1 *:174] 19:S Primary replied to PING, replication can continue...
2025-07-08T13:28:58.464849292Z I0708 15:28:58.464740 1 *:174] 19:S Partial resynchronization not possible (no cached primary)
2025-07-08T13:28:58.465094252Z I0708 15:28:58.465007 1 *:174] 19:S PSYNC is not possible, initialize RDB channel.
2025-07-08T13:28:59.407859535Z I0708 15:28:59.407617 1 *:174] 19:S Shard id bb579abc9e4a814cf4f719203a1eefa38b88e65c update request for node id 07e99ff38fdee190a2f0b4a5178206b408da8765 diverges from existing primary shard id 7d1495a9919f372c9217bef2bd1ce29bc0ad120b, rejecting!
2025-07-08T13:29:02.683277087Z I0708 15:29:02.683095 1 *:174] 19:S Trying a partial resynchronization using main channel (request c50b0358e5373d5d95ce3a7e4a584479058c0cb9:19523487634).
2025-07-08T13:29:02.684388870Z I0708 15:29:02.684188 1 *:174] 19:S Primary <-> REPLICA sync: Primary accepted a Partial Resynchronization, RDB load in background.
2025-07-08T13:29:02.758361483Z I0708 15:29:02.758127 1 *:174] 19:S PRIMARY <-> REPLICA sync: receiving streamed RDB from primary with EOF to parser
2025-07-08T13:29:02.758410203Z I0708 15:29:02.758182 1 *:174] 19:S PRIMARY <-> REPLICA sync: Loading DB in memory
2025-07-08T13:29:02.758415202Z I0708 15:29:02.758241 1 *:174] 19:S Loading RDB produced by Valkey version 8.1.0
2025-07-08T13:29:02.758419792Z I0708 15:29:02.758249 1 *:174] 19:S RDB age 0 seconds
2025-07-08T13:29:02.758424230Z I0708 15:29:02.758255 1 *:174] 19:S RDB memory usage when created 1414.87 Mb
2025-07-08T13:29:03.888819252Z I0708 15:29:03.888037 1 redisnode.go:293] Redis is loading the dataset in memory, passing liveness
2025-07-08T13:29:04.862690765Z I0708 15:29:04.862531 1 redisnode.go:293] Redis is loading the dataset in memory, passing liveness
2025-07-08T13:29:05.088333371Z I0708 15:29:05.088134 1 *:142] === VALKEY BUG REPORT START: Cut & paste starting from here ===
2025-07-08T13:29:05.088366957Z W0708 15:29:05.088171 1 *:176] 19:S === ASSERTION FAILED ===
2025-07-08T13:29:05.088372947Z W0708 15:29:05.088182 1 *:176] 19:S ==> cluster_legacy.c:1388 '!link->node' is not true
2025-07-08T13:29:05.088376951Z I0708 15:29:05.088190 1 *:142] ------ STACK TRACE ------
2025-07-08T13:29:05.091768479Z I0708 15:29:05.091547 1 *:142] 22 bio_lazy_free
2025-07-08T13:29:05.091789066Z I0708 15:29:05.091596 1 *:142] /lib64/libc.so.6(+0xa3c1e)[0x7f489f829c1e]
2025-07-08T13:29:05.091792530Z I0708 15:29:05.091602 1 *:142] /lib64/libc.so.6(pthread_cond_wait+0x12a)[0x7f489f82c548]
2025-07-08T13:29:05.091999048Z I0708 15:29:05.091834 1 *:142] redis-server *:6380 [cluster](bioProcessBackgroundJobs+0x18e)[0x53b61e]
2025-07-08T13:29:05.092014633Z I0708 15:29:05.091844 1 *:142] /lib64/libc.so.6(+0xa6f6c)[0x7f489f82cf6c]
2025-07-08T13:29:05.092017519Z I0708 15:29:05.091882 1 *:142] /lib64/libc.so.6(clone+0x42)[0x7f489f8b4180]
2025-07-08T13:29:05.092020906Z I0708 15:29:05.091886 1 *:142] 20 bio_close_file
2025-07-08T13:29:05.092023953Z I0708 15:29:05.091909 1 *:142] /lib64/libc.so.6(+0xa3c1e)[0x7f489f829c1e]
2025-07-08T13:29:05.092034084Z I0708 15:29:05.091970 1 *:142] /lib64/libc.so.6(pthread_cond_wait+0x12a)[0x7f489f82c548]
2025-07-08T13:29:05.092357758Z I0708 15:29:05.092070 1 *:142] redis-server *:6380 [cluster](bioProcessBackgroundJobs+0x18e)[0x53b61e]
2025-07-08T13:29:05.092362298Z I0708 15:29:05.092137 1 *:142] /lib64/libc.so.6(+0xa6f6c)[0x7f489f82cf6c]
2025-07-08T13:29:05.092364822Z I0708 15:29:05.092158 1 *:142] /lib64/libc.so.6(clone+0x42)[0x7f489f8b4180]
2025-07-08T13:29:05.092367999Z I0708 15:29:05.092162 1 *:142] 21 bio_aof
2025-07-08T13:29:05.092370570Z I0708 15:29:05.092273 1 *:142] /lib64/libc.so.6(+0xa3c1e)[0x7f489f829c1e]
2025-07-08T13:29:05.092373089Z I0708 15:29:05.092278 1 *:142] /lib64/libc.so.6(pthread_cond_wait+0x12a)[0x7f489f82c548]
2025-07-08T13:29:05.092461498Z I0708 15:29:05.092376 1 *:142] redis-server *:6380 [cluster](bioProcessBackgroundJobs+0x18e)[0x53b61e]
2025-07-08T13:29:05.092494917Z I0708 15:29:05.092403 1 *:142] /lib64/libc.so.6(+0xa6f6c)[0x7f489f82cf6c]
2025-07-08T13:29:05.092805479Z I0708 15:29:05.092595 1 *:142] /lib64/libc.so.6(clone+0x42)[0x7f489f8b4180]
2025-07-08T13:29:05.092818062Z I0708 15:29:05.092608 1 *:142] 19 redis-server *
2025-07-08T13:29:05.092823218Z I0708 15:29:05.092614 1 *:142] redis-server *:6380 [cluster](setClusterNodeToInboundClusterLink+0xc4)[0x51b7c4]
2025-07-08T13:29:05.092840424Z I0708 15:29:05.092684 1 *:142] redis-server *:6380 [cluster](clusterProcessPacket+0x4b3)[0x5293c3]
2025-07-08T13:29:05.092873185Z I0708 15:29:05.092796 1 *:142] redis-server *:6380 [cluster](clusterReadHandler+0x129)[0x52a379]
2025-07-08T13:29:05.093022769Z I0708 15:29:05.092930 1 *:142] redis-server *:6380 [cluster][0x5afc2e]
2025-07-08T13:29:05.093067912Z I0708 15:29:05.093019 1 *:142] redis-server *:6380 [cluster][0x5b9eff]
2025-07-08T13:29:05.093200180Z I0708 15:29:05.093117 1 *:142] redis-server *:6380 [cluster][0x4684a2]
2025-07-08T13:29:05.093253834Z I0708 15:29:05.093209 1 *:142] redis-server *:6380 [cluster](processEventsWhileBlocked+0xa5)[0x4a5965]
2025-07-08T13:29:05.093351870Z I0708 15:29:05.093299 1 *:142] redis-server *:6380 [cluster](rdbLoadProgressCallback+0xa0)[0x4ce3c0]
2025-07-08T13:29:05.093439479Z I0708 15:29:05.093357 1 *:142] redis-server *:6380 [cluster][0x4b6748]
2025-07-08T13:29:05.093524406Z I0708 15:29:05.093462 1 *:142] redis-server *:6380 [cluster](rdbGenericLoadStringObject+0x91)[0x4c6041]
2025-07-08T13:29:05.093588413Z I0708 15:29:05.093533 1 *:142] redis-server *:6380 [cluster](rdbLoadObject+0x1267)[0x4cc907]
2025-07-08T13:29:05.093676978Z I0708 15:29:05.093625 1 *:142] redis-server *:6380 [cluster](rdbLoadRioWithLoadingCtx+0x8b3)[0x4cf003]
2025-07-08T13:29:05.093760532Z I0708 15:29:05.093717 1 *:142] redis-server *:6380 [cluster](readSyncBulkPayload+0x290)[0x4c04e0]
2025-07-08T13:29:05.093897367Z I0708 15:29:05.093842 1 *:142] redis-server *:6380 [cluster][0x5afc2e]
2025-07-08T13:29:05.094007517Z I0708 15:29:05.093943 1 *:142] redis-server *:6380 [cluster][0x5b9eff]
2025-07-08T13:29:05.094089642Z I0708 15:29:05.094049 1 *:142] redis-server *:6380 [cluster][0x5bac7c]
2025-07-08T13:29:05.094248493Z I0708 15:29:05.094132 1 *:142] redis-server *:6380 [cluster](beforeSleep+0x66)[0x4707a6]
2025-07-08T13:29:05.094263484Z I0708 15:29:05.094238 1 *:142] redis-server *:6380 [cluster](aeMain+0x2f)[0x46872f]
2025-07-08T13:29:05.094352783Z I0708 15:29:05.094304 1 *:142] redis-server *:6380 [cluster](main+0x4a7)[0x45e1d7]
2025-07-08T13:29:05.094362548Z I0708 15:29:05.094343 1 *:142] /lib64/libc.so.6(+0x40e6c)[0x7f489f7c6e6c]
2025-07-08T13:29:05.094414392Z I0708 15:29:05.094376 1 *:142] /lib64/libc.so.6(__libc_start_main+0x87)[0x7f489f7c6f35]
2025-07-08T13:29:05.094478323Z I0708 15:29:05.094423 1 *:142] redis-server *:6380 [cluster](_start+0x21)[0x45e801]
2025-07-08T13:29:05.094522877Z I0708 15:29:05.094480 1 *:142] 4/4 expected stacktraces.
2025-07-08T13:29:05.094578614Z I0708 15:29:05.094533 1 *:142] ------ STACK TRACE DONE ------
2025-07-08T13:29:05.094582489Z I0708 15:29:05.094545 1 *:142] ------ INFO OUTPUT ------
2025-07-08T13:29:05.095933752Z I0708 15:29:05.095809 1 *:142] # Server
2025-07-08T13:29:05.095943986Z I0708 15:29:05.095830 1 *:142] redis_version:7.2.4
2025-07-08T13:29:05.095946931Z I0708 15:29:05.095836 1 *:142] server_name:valkey
2025-07-08T13:29:05.095949526Z I0708 15:29:05.095842 1 *:142] valkey_version:8.0.3
2025-07-08T13:29:05.095952028Z I0708 15:29:05.095846 1 *:142] redis_git_sha1:00000000
2025-07-08T13:29:05.095964366Z I0708 15:29:05.095851 1 *:142] redis_git_dirty:0
2025-07-08T13:29:05.095982840Z I0708 15:29:05.095857 1 *:142] redis_build_id:e5442683b4ff58ce
2025-07-08T13:29:05.095987076Z I0708 15:29:05.095862 1 *:142] server_mode:cluster
2025-07-08T13:29:05.095991368Z I0708 15:29:05.095868 1 *:142] os:Linux 6.4.0-150600.23.42-default x86_64
2025-07-08T13:29:05.095995338Z I0708 15:29:05.095873 1 *:142] arch_bits:64
2025-07-08T13:29:05.095998976Z I0708 15:29:05.095879 1 *:142] monotonic_clock:POSIX clock_gettime
2025-07-08T13:29:05.096002768Z I0708 15:29:05.095885 1 *:142] multiplexing_api:epoll
2025-07-08T13:29:05.096010624Z I0708 15:29:05.095890 1 *:142] gcc_version:13.3.0
2025-07-08T13:29:05.096014659Z I0708 15:29:05.095896 1 *:142] process_id:19
2025-07-08T13:29:05.096018506Z I0708 15:29:05.095902 1 *:142] process_supervised:no
2025-07-08T13:29:05.096045253Z I0708 15:29:05.095908 1 *:142] run_id:cf2a1fcc13cad10ddaccdb77aa54630805d824f1
2025-07-08T13:29:05.096055915Z I0708 15:29:05.095935 1 *:142] tcp_port:6380
2025-07-08T13:29:05.096059846Z I0708 15:29:05.095941 1 *:142] server_time_usec:1751981345087147
2025-07-08T13:29:05.096063118Z I0708 15:29:05.095946 1 *:142] uptime_in_seconds:31
2025-07-08T13:29:05.096072716Z I0708 15:29:05.095952 1 *:142] uptime_in_days:0
2025-07-08T13:29:05.096079000Z I0708 15:29:05.095958 1 *:142] hz:10
2025-07-08T13:29:05.096082819Z I0708 15:29:05.095964 1 *:142] configured_hz:10
2025-07-08T13:29:05.096086350Z I0708 15:29:05.095969 1 *:142] lru_clock:7150878
2025-07-08T13:29:05.096090029Z I0708 15:29:05.095975 1 *:142] executable:/redis-server
2025-07-08T13:29:05.096094182Z I0708 15:29:05.095980 1 *:142] config_file:/redis-conf/redis.conf
2025-07-08T13:29:05.096098207Z I0708 15:29:05.095986 1 *:142] io_threads_active:0
2025-07-08T13:29:05.096107690Z I0708 15:29:05.095991 1 *:142] availability_zone:
2025-07-08T13:29:05.096112026Z I0708 15:29:05.095997 1 *:142] listener2:name=tls,bind=*,bind=-::*,port=6380
2025-07-08T13:29:05.096115818Z I0708 15:29:05.096002 1 *:142] # Clients
2025-07-08T13:29:05.096119483Z I0708 15:29:05.096007 1 *:142] connected_clients:1
2025-07-08T13:29:05.096123173Z I0708 15:29:05.096013 1 *:142] cluster_connections:24
2025-07-08T13:29:05.096141991Z I0708 15:29:05.096019 1 *:142] maxclients:8192
2025-07-08T13:29:05.096146016Z I0708 15:29:05.096035 1 *:142] client_recent_max_input_buffer:0
2025-07-08T13:29:05.096148447Z I0708 15:29:05.096041 1 *:142] client_recent_max_output_buffer:0
2025-07-08T13:29:05.096150975Z I0708 15:29:05.096047 1 *:142] blocked_clients:0
2025-07-08T13:29:05.096153357Z I0708 15:29:05.096052 1 *:142] tracking_clients:0
2025-07-08T13:29:05.096155774Z I0708 15:29:05.096058 1 *:142] pubsub_clients:0
2025-07-08T13:29:05.096164261Z I0708 15:29:05.096063 1 *:142] watching_clients:0
2025-07-08T13:29:05.096167141Z I0708 15:29:05.096069 1 *:142] clients_in_timeout_table:0
2025-07-08T13:29:05.096169579Z I0708 15:29:05.096075 1 *:142] total_watched_keys:0
2025-07-08T13:29:05.096172030Z I0708 15:29:05.096081 1 *:142] total_blocking_keys:0
2025-07-08T13:29:05.096174568Z I0708 15:29:05.096086 1 *:142] total_blocking_keys_on_nokey:0
2025-07-08T13:29:05.096203361Z I0708 15:29:05.096091 1 *:142] # Memory
2025-07-08T13:29:05.096206221Z I0708 15:29:05.096098 1 *:142] used_memory:179551640
2025-07-08T13:29:05.096208675Z I0708 15:29:05.096103 1 *:142] used_memory_human:171.23M
2025-07-08T13:29:05.096211055Z I0708 15:29:05.096109 1 *:142] used_memory_rss:185344000
2025-07-08T13:29:05.096213429Z I0708 15:29:05.096115 1 *:142] used_memory_rss_human:176.76M
2025-07-08T13:29:05.096215833Z I0708 15:29:05.096120 1 *:142] used_memory_peak:179551640
2025-07-08T13:29:05.096218235Z I0708 15:29:05.096126 1 *:142] used_memory_peak_human:171.23M
2025-07-08T13:29:05.096224682Z I0708 15:29:05.096133 1 *:142] used_memory_peak_perc:100.01%
2025-07-08T13:29:05.096227163Z I0708 15:29:05.096139 1 *:142] used_memory_overhead:2722112
2025-07-08T13:29:05.096229532Z I0708 15:29:05.096145 1 *:142] used_memory_startup:2690808
2025-07-08T13:29:05.096232016Z I0708 15:29:05.096151 1 *:142] used_memory_dataset:176829528
2025-07-08T13:29:05.096237855Z I0708 15:29:05.096156 1 *:142] used_memory_dataset_perc:99.98%
2025-07-08T13:29:05.096240504Z I0708 15:29:05.096162 1 *:142] allocator_allocated:172144928
2025-07-08T13:29:05.096242874Z I0708 15:29:05.096167 1 *:142] allocator_active:172363776
2025-07-08T13:29:05.096245139Z I0708 15:29:05.096173 1 *:142] allocator_resident:183177216
2025-07-08T13:29:05.096261596Z I0708 15:29:05.096178 1 *:142] allocator_muzzy:0
2025-07-08T13:29:05.096297464Z I0708 15:29:05.096195 1 *:142] total_system_memory:168782573568
2025-07-08T13:29:05.096304317Z I0708 15:29:05.096216 1 *:142] total_system_memory_human:157.19G
2025-07-08T13:29:05.096307318Z I0708 15:29:05.096222 1 *:142] used_memory_lua:31744
2025-07-08T13:29:05.096319320Z I0708 15:29:05.096227 1 *:142] used_memory_vm_eval:31744
2025-07-08T13:29:05.096322423Z I0708 15:29:05.096232 1 *:142] used_memory_lua_human:31.00K
2025-07-08T13:29:05.096324886Z I0708 15:29:05.096238 1 *:142] used_memory_scripts_eval:0
2025-07-08T13:29:05.096327266Z I0708 15:29:05.096244 1 *:142] number_of_cached_scripts:0
2025-07-08T13:29:05.096329775Z I0708 15:29:05.096266 1 *:142] number_of_functions:0
2025-07-08T13:29:05.096333326Z I0708 15:29:05.096271 1 *:142] number_of_libraries:0
2025-07-08T13:29:05.096337029Z I0708 15:29:05.096277 1 *:142] used_memory_vm_functions:33792
2025-07-08T13:29:05.096340720Z I0708 15:29:05.096283 1 *:142] used_memory_vm_total:65536
2025-07-08T13:29:05.096351253Z I0708 15:29:05.096288 1 *:142] used_memory_vm_total_human:64.00K
2025-07-08T13:29:05.096355737Z I0708 15:29:05.096294 1 *:142] used_memory_functions:184
2025-07-08T13:29:05.096359716Z I0708 15:29:05.096299 1 *:142] used_memory_scripts:184
2025-07-08T13:29:05.096363396Z I0708 15:29:05.096305 1 *:142] used_memory_scripts_human:184B
2025-07-08T13:29:05.096367228Z I0708 15:29:05.096311 1 *:142] maxmemory:5046586572
2025-07-08T13:29:05.096371090Z I0708 15:29:05.096316 1 *:142] maxmemory_human:4.70G
2025-07-08T13:29:05.096374800Z I0708 15:29:05.096322 1 *:142] maxmemory_policy:noeviction
2025-07-08T13:29:05.096378858Z I0708 15:29:05.096329 1 *:142] allocator_frag_ratio:1.00
2025-07-08T13:29:05.096391295Z I0708 15:29:05.096334 1 *:142] allocator_frag_bytes:218848
2025-07-08T13:29:05.096402753Z I0708 15:29:05.096340 1 *:142] allocator_rss_ratio:1.06
2025-07-08T13:29:05.096408349Z I0708 15:29:05.096345 1 *:142] allocator_rss_bytes:10813440
2025-07-08T13:29:05.096411090Z I0708 15:29:05.096350 1 *:142] rss_overhead_ratio:1.01
2025-07-08T13:29:05.096413652Z I0708 15:29:05.096356 1 *:142] rss_overhead_bytes:2166784
2025-07-08T13:29:05.096416293Z I0708 15:29:05.096363 1 *:142] mem_fragmentation_ratio:1.08
2025-07-08T13:29:05.096423999Z I0708 15:29:05.096369 1 *:142] mem_fragmentation_bytes:13351608
2025-07-08T13:29:05.096426758Z I0708 15:29:05.096375 1 *:142] mem_not_counted_for_evict:0
2025-07-08T13:29:05.096429259Z I0708 15:29:05.096381 1 *:142] mem_replication_backlog:0
2025-07-08T13:29:05.096431925Z I0708 15:29:05.096386 1 *:142] mem_total_replication_buffers:0
2025-07-08T13:29:05.096437133Z I0708 15:29:05.096392 1 *:142] mem_clients_slaves:0
2025-07-08T13:29:05.096443368Z I0708 15:29:05.096397 1 *:142] mem_clients_normal:1920
2025-07-08T13:29:05.096446155Z I0708 15:29:05.096403 1 *:142] mem_cluster_links:29200
2025-07-08T13:29:05.096448587Z I0708 15:29:05.096409 1 *:142] mem_aof_buffer:0
2025-07-08T13:29:05.096454570Z I0708 15:29:05.096415 1 *:142] mem_allocator:jemalloc-5.3.0
2025-07-08T13:29:05.096457268Z I0708 15:29:05.096421 1 *:142] mem_overhead_db_hashtable_rehashing:0
2025-07-08T13:29:05.096459542Z I0708 15:29:05.096427 1 *:142] active_defrag_running:0
2025-07-08T13:29:05.096462131Z I0708 15:29:05.096433 1 *:142] lazyfree_pending_objects:0
2025-07-08T13:29:05.096467925Z I0708 15:29:05.096438 1 *:142] lazyfreed_objects:0
2025-07-08T13:29:05.096474490Z I0708 15:29:05.096445 1 *:142] # Persistence
2025-07-08T13:29:05.096477222Z I0708 15:29:05.096450 1 *:142] loading:1
2025-07-08T13:29:05.096479497Z I0708 15:29:05.096455 1 *:142] async_loading:0
2025-07-08T13:29:05.096485330Z I0708 15:29:05.096461 1 *:142] current_cow_peak:0
2025-07-08T13:29:05.096492328Z I0708 15:29:05.096466 1 *:142] current_cow_size:0
2025-07-08T13:29:05.096496611Z I0708 15:29:05.096472 1 *:142] current_cow_size_age:0
2025-07-08T13:29:05.096503747Z I0708 15:29:05.096478 1 *:142] current_fork_perc:0.00
2025-07-08T13:29:05.096506699Z I0708 15:29:05.096483 1 *:142] current_save_keys_processed:0
2025-07-08T13:29:05.096512803Z I0708 15:29:05.096488 1 *:142] current_save_keys_total:0
2025-07-08T13:29:05.096519655Z I0708 15:29:05.096494 1 *:142] rdb_changes_since_last_save:0
2025-07-08T13:29:05.096522513Z I0708 15:29:05.096499 1 *:142] rdb_bgsave_in_progress:0
2025-07-08T13:29:05.096528219Z I0708 15:29:05.096505 1 *:142] rdb_last_save_time:1751981314
2025-07-08T13:29:05.096534126Z I0708 15:29:05.096510 1 *:142] rdb_last_bgsave_status:ok
2025-07-08T13:29:05.096536754Z I0708 15:29:05.096516 1 *:142] rdb_last_bgsave_time_sec:-1
2025-07-08T13:29:05.096542642Z I0708 15:29:05.096522 1 *:142] rdb_current_bgsave_time_sec:-1
2025-07-08T13:29:05.096548807Z I0708 15:29:05.096528 1 *:142] rdb_saves:0
2025-07-08T13:29:05.096556339Z I0708 15:29:05.096534 1 *:142] rdb_last_cow_size:0
2025-07-08T13:29:05.096607463Z I0708 15:29:05.096553 1 *:142] rdb_last_load_keys_expired:0
2025-07-08T13:29:05.096614134Z I0708 15:29:05.096570 1 *:142] rdb_last_load_keys_loaded:53884
2025-07-08T13:29:05.096618027Z I0708 15:29:05.096580 1 *:142] aof_enabled:0
2025-07-08T13:29:05.096622174Z I0708 15:29:05.096585 1 *:142] aof_rewrite_in_progress:0
2025-07-08T13:29:05.096632343Z I0708 15:29:05.096590 1 *:142] aof_rewrite_scheduled:0
2025-07-08T13:29:05.096635720Z I0708 15:29:05.096596 1 *:142] aof_last_rewrite_time_sec:-1
2025-07-08T13:29:05.096638934Z I0708 15:29:05.096601 1 *:142] aof_current_rewrite_time_sec:-1
2025-07-08T13:29:05.096643018Z I0708 15:29:05.096607 1 *:142] aof_last_bgrewrite_status:ok
2025-07-08T13:29:05.096652167Z I0708 15:29:05.096612 1 *:142] aof_rewrites:0
2025-07-08T13:29:05.096656445Z I0708 15:29:05.096618 1 *:142] aof_rewrites_consecutive_failures:0
2025-07-08T13:29:05.096664940Z I0708 15:29:05.096624 1 *:142] aof_last_write_status:ok
2025-07-08T13:29:05.096670610Z I0708 15:29:05.096631 1 *:142] aof_last_cow_size:0
2025-07-08T13:29:05.096673555Z I0708 15:29:05.096636 1 *:142] module_fork_in_progress:0
2025-07-08T13:29:05.096681047Z I0708 15:29:05.096642 1 *:142] module_fork_last_cow_size:0
2025-07-08T13:29:05.096690829Z I0708 15:29:05.096647 1 *:142] loading_start_time:1751981342
2025-07-08T13:29:05.096693447Z I0708 15:29:05.096653 1 *:142] loading_total_bytes:0
2025-07-08T13:29:05.096707093Z I0708 15:29:05.096658 1 *:142] loading_rdb_used_mem:1483595224
2025-07-08T13:29:05.096709771Z I0708 15:29:05.096663 1 *:142] loading_loaded_bytes:146798903
2025-07-08T13:29:05.096715814Z I0708 15:29:05.096669 1 *:142] loading_loaded_perc:9.89
2025-07-08T13:29:05.096730893Z I0708 15:29:05.096675 1 *:142] loading_eta_seconds:27
2025-07-08T13:29:05.096734111Z I0708 15:29:05.096680 1 *:142] # Stats
2025-07-08T13:29:05.096736805Z I0708 15:29:05.096685 1 *:142] total_connections_received:20
2025-07-08T13:29:05.096742613Z I0708 15:29:05.096691 1 *:142] total_commands_processed:94
2025-07-08T13:29:05.096745464Z I0708 15:29:05.096697 1 *:142] instantaneous_ops_per_sec:0
2025-07-08T13:29:05.096747998Z I0708 15:29:05.096702 1 *:142] total_net_input_bytes:146802567
2025-07-08T13:29:05.096756835Z I0708 15:29:05.096707 1 *:142] total_net_output_bytes:68735
2025-07-08T13:29:05.096759658Z I0708 15:29:05.096713 1 *:142] total_net_repl_input_bytes:146798950
2025-07-08T13:29:05.096761962Z I0708 15:29:05.096719 1 *:142] total_net_repl_output_bytes:0
2025-07-08T13:29:05.096764330Z I0708 15:29:05.096724 1 *:142] instantaneous_input_kbps:0.00
2025-07-08T13:29:05.096766801Z I0708 15:29:05.096730 1 *:142] instantaneous_output_kbps:0.00
2025-07-08T13:29:05.096775774Z I0708 15:29:05.096736 1 *:142] instantaneous_input_repl_kbps:0.00
2025-07-08T13:29:05.096778496Z I0708 15:29:05.096741 1 *:142] instantaneous_output_repl_kbps:0.00
2025-07-08T13:29:05.096784064Z I0708 15:29:05.096747 1 *:142] rejected_connections:0
2025-07-08T13:29:05.096786807Z I0708 15:29:05.096752 1 *:142] sync_full:0
2025-07-08T13:29:05.096789121Z I0708 15:29:05.096758 1 *:142] sync_partial_ok:0
2025-07-08T13:29:05.096808983Z I0708 15:29:05.096765 1 *:142] sync_partial_err:0
2025-07-08T13:29:05.096812041Z I0708 15:29:05.096772 1 *:142] expired_keys:0
2025-07-08T13:29:05.096817660Z I0708 15:29:05.096776 1 *:142] expired_stale_perc:0.00
2025-07-08T13:29:05.096820285Z I0708 15:29:05.096782 1 *:142] expired_time_cap_reached_count:0
2025-07-08T13:29:05.096822892Z I0708 15:29:05.096787 1 *:142] expire_cycle_cpu_milliseconds:0
2025-07-08T13:29:05.096828206Z I0708 15:29:05.096792 1 *:142] evicted_keys:0
2025-07-08T13:29:05.096834192Z I0708 15:29:05.096798 1 *:142] evicted_clients:0
2025-07-08T13:29:05.096836866Z I0708 15:29:05.096804 1 *:142] evicted_scripts:0
2025-07-08T13:29:05.096839207Z I0708 15:29:05.096809 1 *:142] total_eviction_exceeded_time:0
2025-07-08T13:29:05.096842012Z I0708 15:29:05.096814 1 *:142] current_eviction_exceeded_time:0
2025-07-08T13:29:05.096847835Z I0708 15:29:05.096820 1 *:142] keyspace_hits:0
2025-07-08T13:29:05.096855742Z I0708 15:29:05.096825 1 *:142] keyspace_misses:0
2025-07-08T13:29:05.096858532Z I0708 15:29:05.096831 1 *:142] pubsub_channels:0
2025-07-08T13:29:05.096860934Z I0708 15:29:05.096837 1 *:142] pubsub_patterns:0
2025-07-08T13:29:05.096866371Z I0708 15:29:05.096842 1 *:142] pubsubshard_channels:0
2025-07-08T13:29:05.096872204Z I0708 15:29:05.096848 1 *:142] latest_fork_usec:0
2025-07-08T13:29:05.096878068Z I0708 15:29:05.096853 1 *:142] total_forks:0
2025-07-08T13:29:05.096880693Z I0708 15:29:05.096858 1 *:142] migrate_cached_sockets:0
2025-07-08T13:29:05.096883141Z I0708 15:29:05.096863 1 *:142] slave_expires_tracked_keys:0
2025-07-08T13:29:05.096890449Z I0708 15:29:05.096869 1 *:142] active_defrag_hits:0
2025-07-08T13:29:05.096892976Z I0708 15:29:05.096874 1 *:142] active_defrag_misses:0
2025-07-08T13:29:05.096898578Z I0708 15:29:05.096880 1 *:142] active_defrag_key_hits:0
2025-07-08T13:29:05.096901263Z I0708 15:29:05.096885 1 *:142] active_defrag_key_misses:0
2025-07-08T13:29:05.096907010Z I0708 15:29:05.096891 1 *:142] total_active_defrag_time:0
2025-07-08T13:29:05.096909673Z I0708 15:29:05.096896 1 *:142] current_active_defrag_time:0
2025-07-08T13:29:05.096915130Z I0708 15:29:05.096902 1 *:142] tracking_total_keys:0
2025-07-08T13:29:05.096933570Z I0708 15:29:05.096907 1 *:142] tracking_total_items:0
2025-07-08T13:29:05.096936843Z I0708 15:29:05.096913 1 *:142] tracking_total_prefixes:0
2025-07-08T13:29:05.096939236Z I0708 15:29:05.096918 1 *:142] unexpected_error_replies:0
2025-07-08T13:29:05.096944824Z I0708 15:29:05.096924 1 *:142] total_error_replies:2
2025-07-08T13:29:05.096947450Z I0708 15:29:05.096930 1 *:142] dump_payload_sanitizations:0
2025-07-08T13:29:05.096962882Z I0708 15:29:05.096935 1 *:142] total_reads_processed:496
2025-07-08T13:29:05.096966049Z I0708 15:29:05.096941 1 *:142] total_writes_processed:96
2025-07-08T13:29:05.096968512Z I0708 15:29:05.096947 1 *:142] io_threaded_reads_processed:0
2025-07-08T13:29:05.096973942Z I0708 15:29:05.096953 1 *:142] io_threaded_writes_processed:0
2025-07-08T13:29:05.096976523Z I0708 15:29:05.096958 1 *:142] io_threaded_freed_objects:0
2025-07-08T13:29:05.096986429Z I0708 15:29:05.096964 1 *:142] io_threaded_poll_processed:0
2025-07-08T13:29:05.096989303Z I0708 15:29:05.096969 1 *:142] io_threaded_total_prefetch_batches:0
2025-07-08T13:29:05.096995047Z I0708 15:29:05.096975 1 *:142] io_threaded_total_prefetch_entries:0
2025-07-08T13:29:05.096997570Z I0708 15:29:05.096981 1 *:142] client_query_buffer_limit_disconnections:0
2025-07-08T13:29:05.097006591Z I0708 15:29:05.096986 1 *:142] client_output_buffer_limit_disconnections:0
2025-07-08T13:29:05.097013184Z I0708 15:29:05.096992 1 *:142] reply_buffer_shrinks:17
2025-07-08T13:29:05.097020517Z I0708 15:29:05.096997 1 *:142] reply_buffer_expands:5
2025-07-08T13:29:05.097024324Z I0708 15:29:05.097003 1 *:142] eventloop_cycles:1557
2025-07-08T13:29:05.097030401Z I0708 15:29:05.097008 1 *:142] eventloop_duration_sum:359858
2025-07-08T13:29:05.097033274Z I0708 15:29:05.097013 1 *:142] eventloop_duration_cmd_sum:5348
2025-07-08T13:29:05.097038483Z I0708 15:29:05.097019 1 *:142] instantaneous_eventloop_cycles_per_sec:45
2025-07-08T13:29:05.097041260Z I0708 15:29:05.097025 1 *:142] instantaneous_eventloop_duration_usec:162
2025-07-08T13:29:05.097046932Z I0708 15:29:05.097030 1 *:142] acl_access_denied_auth:0
2025-07-08T13:29:05.097049472Z I0708 15:29:05.097036 1 *:142] acl_access_denied_cmd:0
2025-07-08T13:29:05.097054818Z I0708 15:29:05.097042 1 *:142] acl_access_denied_key:0
2025-07-08T13:29:05.097060352Z I0708 15:29:05.097047 1 *:142] acl_access_denied_channel:0
2025-07-08T13:29:05.097074482Z I0708 15:29:05.097052 1 *:142] # Replication
2025-07-08T13:29:05.097077523Z I0708 15:29:05.097058 1 *:142] role:slave
2025-07-08T13:29:05.097080160Z I0708 15:29:05.097064 1 *:142] master_host:192.168.26.185
2025-07-08T13:29:05.097085727Z I0708 15:29:05.097069 1 *:142] master_port:6380
2025-07-08T13:29:05.097091428Z I0708 15:29:05.097075 1 *:142] master_link_status:down
2025-07-08T13:29:05.097094070Z I0708 15:29:05.097081 1 *:142] master_last_io_seconds_ago:-1
2025-07-08T13:29:05.097099489Z I0708 15:29:05.097086 1 *:142] master_sync_in_progress:1
2025-07-08T13:29:05.097105249Z I0708 15:29:05.097092 1 *:142] slave_read_repl_offset:1
2025-07-08T13:29:05.097110808Z I0708 15:29:05.097097 1 *:142] slave_repl_offset:1
2025-07-08T13:29:05.097120601Z I0708 15:29:05.097102 1 *:142] replicas_repl_buffer_size:1023200
2025-07-08T13:29:05.097126570Z I0708 15:29:05.097108 1 *:142] replicas_repl_buffer_peak:1023200
2025-07-08T13:29:05.097132006Z I0708 15:29:05.097114 1 *:142] master_sync_total_bytes:0
2025-07-08T13:29:05.097138946Z I0708 15:29:05.097120 1 *:142] master_sync_read_bytes:0
2025-07-08T13:29:05.097142355Z I0708 15:29:05.097125 1 *:142] master_sync_left_bytes:0
2025-07-08T13:29:05.097148180Z I0708 15:29:05.097131 1 *:142] master_sync_perc:0.00
2025-07-08T13:29:05.097155572Z I0708 15:29:05.097137 1 *:142] master_sync_last_io_seconds_ago:3
2025-07-08T13:29:05.097161330Z I0708 15:29:05.097142 1 *:142] master_link_down_since_seconds:-1
2025-07-08T13:29:05.097167206Z I0708 15:29:05.097148 1 *:142] slave_priority:100
2025-07-08T13:29:05.097174982Z I0708 15:29:05.097153 1 *:142] slave_read_only:1
2025-07-08T13:29:05.097195332Z I0708 15:29:05.097159 1 *:142] replica_announced:1
2025-07-08T13:29:05.097202647Z I0708 15:29:05.097165 1 *:142] connected_slaves:0
2025-07-08T13:29:05.097207863Z I0708 15:29:05.097170 1 *:142] replicas_waiting_psync:0
2025-07-08T13:29:05.097210826Z I0708 15:29:05.097176 1 *:142] master_failover_state:no-failover
2025-07-08T13:29:05.097213707Z I0708 15:29:05.097181 1 *:142] master_replid:ae4654610194716e5acba589632104e58bf3a3fe
2025-07-08T13:29:05.097221584Z I0708 15:29:05.097186 1 *:142] master_replid2:0000000000000000000000000000000000000000
2025-07-08T13:29:05.097224436Z I0708 15:29:05.097192 1 *:142] master_repl_offset:0
2025-07-08T13:29:05.097230403Z I0708 15:29:05.097198 1 *:142] second_repl_offset:-1
2025-07-08T13:29:05.097233171Z I0708 15:29:05.097203 1 *:142] repl_backlog_active:0
2025-07-08T13:29:05.097235866Z I0708 15:29:05.097209 1 *:142] repl_backlog_size:134217728
2025-07-08T13:29:05.097248609Z I0708 15:29:05.097214 1 *:142] repl_backlog_first_byte_offset:0
2025-07-08T13:29:05.097251633Z I0708 15:29:05.097220 1 *:142] repl_backlog_histlen:0
2025-07-08T13:29:05.097258198Z I0708 15:29:05.097226 1 *:142] # CPU
2025-07-08T13:29:05.097260840Z I0708 15:29:05.097231 1 *:142] used_cpu_sys:0.706946
2025-07-08T13:29:05.097263188Z I0708 15:29:05.097237 1 *:142] used_cpu_user:1.271686
2025-07-08T13:29:05.097269057Z I0708 15:29:05.097242 1 *:142] used_cpu_sys_children:0.000000
2025-07-08T13:29:05.097275054Z I0708 15:29:05.097248 1 *:142] used_cpu_user_children:0.000000
2025-07-08T13:29:05.097277779Z I0708 15:29:05.097254 1 *:142] used_cpu_sys_main_thread:0.706625
2025-07-08T13:29:05.097283620Z I0708 15:29:05.097260 1 *:142] used_cpu_user_main_thread:1.271110
2025-07-08T13:29:05.097295113Z I0708 15:29:05.097265 1 *:142] # Modules
2025-07-08T13:29:05.097298018Z I0708 15:29:05.097271 1 *:142] # Commandstats
2025-07-08T13:29:05.097304724Z I0708 15:29:05.097276 1 *:142] cmdstat_info:calls=22,usec=591,usec_per_call=26.86,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097323266Z I0708 15:29:05.097282 1 *:142] cmdstat_slowlog|len:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097326837Z I0708 15:29:05.097287 1 *:142] cmdstat_slowlog|get:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097332687Z I0708 15:29:05.097293 1 *:142] cmdstat_client|setname:calls=2,usec=4,usec_per_call=2.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097336153Z I0708 15:29:05.097298 1 *:142] cmdstat_ping:calls=8,usec=6,usec_per_call=0.75,rejected_calls=2,failed_calls=0
2025-07-08T13:29:05.097338691Z I0708 15:29:05.097304 1 *:142] cmdstat_config|get:calls=1,usec=274,usec_per_call=274.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097341471Z I0708 15:29:05.097309 1 *:142] cmdstat_config|set:calls=19,usec=118,usec_per_call=6.21,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097347132Z I0708 15:29:05.097316 1 *:142] cmdstat_latency|latest:calls=1,usec=3,usec_per_call=3.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097349783Z I0708 15:29:05.097321 1 *:142] cmdstat_latency|histogram:calls=1,usec=32,usec_per_call=32.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097361713Z I0708 15:29:05.097327 1 *:142] cmdstat_cluster|set-config-epoch:calls=1,usec=25,usec_per_call=25.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097364730Z I0708 15:29:05.097333 1 *:142] cmdstat_cluster|info:calls=5,usec=239,usec_per_call=47.80,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097370383Z I0708 15:29:05.097339 1 *:142] cmdstat_cluster|meet:calls=1,usec=22,usec_per_call=22.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097373015Z I0708 15:29:05.097344 1 *:142] cmdstat_cluster|nodes:calls=25,usec=3395,usec_per_call=135.80,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097375655Z I0708 15:29:05.097350 1 *:142] cmdstat_cluster|slots:calls=5,usec=381,usec_per_call=76.20,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097380772Z I0708 15:29:05.097356 1 *:142] cmdstat_cluster|replicate:calls=1,usec=256,usec_per_call=256.00,rejected_calls=0,failed_calls=0
2025-07-08T13:29:05.097383588Z I0708 15:29:05.097361 1 *:142] # Errorstats
2025-07-08T13:29:05.097389304Z I0708 15:29:05.097367 1 *:142] errorstat_LOADING:count=2
2025-07-08T13:29:05.097400354Z I0708 15:29:05.097372 1 *:142] # Latencystats
2025-07-08T13:29:05.097403181Z I0708 15:29:05.097378 1 *:142] # Cluster
2025-07-08T13:29:05.097409022Z I0708 15:29:05.097384 1 *:142] cluster_enabled:1
2025-07-08T13:29:05.097415807Z I0708 15:29:05.097389 1 *:142] # Keyspace
2025-07-08T13:29:05.097421498Z I0708 15:29:05.097394 1 *:142] # Cluster info
2025-07-08T13:29:05.097429057Z I0708 15:29:05.097400 1 *:142] cluster_state:ok
2025-07-08T13:29:05.097431683Z I0708 15:29:05.097405 1 *:142] cluster_slots_assigned:16384
2025-07-08T13:29:05.097451264Z I0708 15:29:05.097411 1 *:142] cluster_slots_ok:16384
2025-07-08T13:29:05.097457544Z I0708 15:29:05.097416 1 *:142] cluster_slots_pfail:0
2025-07-08T13:29:05.097467786Z I0708 15:29:05.097422 1 *:142] cluster_slots_fail:0
2025-07-08T13:29:05.097473372Z I0708 15:29:05.097428 1 *:142] cluster_known_nodes:13
2025-07-08T13:29:05.097476027Z I0708 15:29:05.097434 1 *:142] cluster_size:6
2025-07-08T13:29:05.097478770Z I0708 15:29:05.097440 1 *:142] cluster_current_epoch:44
2025-07-08T13:29:05.097506199Z I0708 15:29:05.097445 1 *:142] cluster_my_epoch:26
2025-07-08T13:29:05.097509782Z I0708 15:29:05.097478 1 *:142] cluster_stats_messages_ping_sent:302
2025-07-08T13:29:05.097512155Z I0708 15:29:05.097484 1 *:142] cluster_stats_messages_pong_sent:364
2025-07-08T13:29:05.097514667Z I0708 15:29:05.097489 1 *:142] cluster_stats_messages_meet_sent:1
2025-07-08T13:29:05.097520672Z I0708 15:29:05.097494 1 *:142] cluster_stats_messages_sent:667
2025-07-08T13:29:05.097523330Z I0708 15:29:05.097501 1 *:142] cluster_stats_messages_ping_received:329
2025-07-08T13:29:05.097525867Z I0708 15:29:05.097507 1 *:142] cluster_stats_messages_pong_received:302
2025-07-08T13:29:05.097531226Z I0708 15:29:05.097513 1 *:142] cluster_stats_messages_meet_received:24
2025-07-08T13:29:05.097533679Z I0708 15:29:05.097518 1 *:142] cluster_stats_messages_fail_received:2
2025-07-08T13:29:05.097560873Z I0708 15:29:05.097524 1 *:142] cluster_stats_messages_received:657
2025-07-08T13:29:05.097564326Z I0708 15:29:05.097537 1 *:142] total_cluster_links_buffer_limit_exceeded:0
2025-07-08T13:29:05.097570602Z I0708 15:29:05.097543 1 *:142] ------ CLUSTER NODES OUTPUT ------
2025-07-08T13:29:05.097577165Z I0708 15:29:05.097549 1 *:142] 3686a09e4eccb67c4ceabbf21ce21c3ce8cd84df 192.168.26.139:6380@16380,,tls-port=6380,shard-id=4e7e00f4fb9a49fec0949a59a3aecc087ff5b135 slave e84dbcaccaeda6f281bdfdf6047cd0384b4d8d4f 0 1751981344000 36 connected
2025-07-08T13:29:05.097580206Z I0708 15:29:05.097556 1 *:142] 5f63c9d2eebe3febbda7fee6cee02b1677761ade 192.168.26.185:6380@16380,,tls-port=6380,shard-id=7d1495a9919f372c9217bef2bd1ce29bc0ad120b master - 0 1751981344000 26 connected 5461-8190
2025-07-08T13:29:05.097636701Z I0708 15:29:05.097562 1 *:142] e7939ebc7d83aebff46de55af12d2c069d7bfcbd 192.168.26.167:6380@16380,,tls-port=6380,shard-id=bcb7a0426b6613a4d4317c803019fa9a6a252f60 slave db3ed3647b20522f2872cbb0ee36d2671986d458 0 1751981344000 29 connected
2025-07-08T13:29:05.097641451Z I0708 15:29:05.097567 1 *:142] db3ed3647b20522f2872cbb0ee36d2671986d458 192.168.159.18:6380@16380,,tls-port=6380,shard-id=bcb7a0426b6613a4d4317c803019fa9a6a252f60 master - 0 1751981344000 29 connected 8191-10921
2025-07-08T13:29:05.097644424Z I0708 15:29:05.097572 1 *:142] e7d3c787d77ce2764cd27c850fe1a24b8cd70f49 192.168.117.20:6380@16380,,tls-port=6380,shard-id=2b19f36b9329a4dcddfea4a391cd4b47835539a8 handshake - 0 0 0 disconnected
2025-07-08T13:29:05.097646868Z I0708 15:29:05.097577 1 *:142] b81e54f8bc2e7d4a091ebc7668d892321da83ffa 192.168.82.89:6380@16380,,tls-port=6380,shard-id=ebdc7bb15ddbfab35771dc9bae40257f0bfc8190 master - 0 1751981344000 35 connected 13653-16383
2025-07-08T13:29:05.097649206Z I0708 15:29:05.097583 1 *:142] 16bad5bdbddde39e918298678300a222870a6e7a 192.168.82.124:6380@16380,,tls-port=6380,shard-id=1ad9099d4c2ec38bce114979774fd55a1d23647c master - 0 1751981344000 42 connected 0-2730
2025-07-08T13:29:05.097651486Z I0708 15:29:05.097588 1 *:142] 529f0f2ab1242ddc83fe1e200c57600d16640d41 192.168.159.35:6380@16380,,tls-port=6380,shard-id=7f6d349525702cb601b660a4765480bf24aa8b30 master - 0 1751981344000 39 connected 2731-5460
2025-07-08T13:29:05.097654371Z I0708 15:29:05.097594 1 *:142] 5a874147b478800c3c5177e0a7c68f1d781977af 192.168.103.40:6380@16380,,tls-port=6380,shard-id=7d1495a9919f372c9217bef2bd1ce29bc0ad120b myself,slave 5f63c9d2eebe3febbda7fee6cee02b1677761ade 0 0 26 connected
2025-07-08T13:29:05.097656629Z I0708 15:29:05.097600 1 *:142] e84dbcaccaeda6f281bdfdf6047cd0384b4d8d4f 192.168.117.30:6380@16380,,tls-port=6380,shard-id=4e7e00f4fb9a49fec0949a59a3aecc087ff5b135 master - 0 1751981344000 36 connected 10922-13652
2025-07-08T13:29:05.097659029Z I0708 15:29:05.097606 1 *:142] 7fe96c396c11bfdc24aa541790395d6b8f7c97fb 192.168.47.240:6380@16380,,tls-port=6380,shard-id=ebdc7bb15ddbfab35771dc9bae40257f0bfc8190 slave b81e54f8bc2e7d4a091ebc7668d892321da83ffa 0 1751981344000 35 connected
2025-07-08T13:29:05.097661662Z I0708 15:29:05.097611 1 *:142] 7aa7b4247fc1a54bbf78a77201ad195c8d87ef7c 192.168.103.11:6380@16380,,tls-port=6380,shard-id=7f6d349525702cb601b660a4765480bf24aa8b30 slave 529f0f2ab1242ddc83fe1e200c57600d16640d41 0 1751981344000 39 connected
2025-07-08T13:29:05.097668004Z I0708 15:29:05.097617 1 *:142] fb16529c28535edea7b59bd12023486c74ad8234 192.168.82.125:6380@16380,,tls-port=6380,shard-id=7d1495a9919f372c9217bef2bd1ce29bc0ad120b slave 5f63c9d2eebe3febbda7fee6cee02b1677761ade 0 1751981344000 26 connected
2025-07-08T13:29:05.097670806Z I0708 15:29:05.097623 1 *:142] ------ CLIENT LIST OUTPUT ------
2025-07-08T13:29:05.100342700Z I0708 15:29:05.100242 1 *:142] id=4 addr=127.0.0.1:49266 laddr=127.0.0.1:6380 fd=13 name=*redacted* age=31 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=1024 obl=0 oll=0 omem=0 tot-mem=1920 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=508 tot-net-out=10436 tot-cmds=18
2025-07-08T13:29:05.100367157Z I0708 15:29:05.100272 1 *:142] ------ MODULES INFO OUTPUT ------
2025-07-08T13:29:05.100370187Z I0708 15:29:05.100280 1 *:142] ------ CONFIG DEBUG OUTPUT ------
2025-07-08T13:29:05.100372802Z I0708 15:29:05.100288 1 *:142] proto-max-bulk-len 512mb
2025-07-08T13:29:05.100375457Z I0708 15:29:05.100293 1 *:142] list-compress-depth 0
2025-07-08T13:29:05.100378005Z I0708 15:29:05.100299 1 *:142] dual-channel-replication-enabled yes
2025-07-08T13:29:05.100380519Z I0708 15:29:05.100305 1 *:142] lazyfree-lazy-user-del yes
2025-07-08T13:29:05.100382900Z I0708 15:29:05.100310 1 *:142] client-query-buffer-limit 1gb
2025-07-08T13:29:05.100385328Z I0708 15:29:05.100315 1 *:142] lazyfree-lazy-user-flush yes
2025-07-08T13:29:05.100387752Z I0708 15:29:05.100321 1 *:142] lazyfree-lazy-eviction yes
2025-07-08T13:29:05.100394715Z I0708 15:29:05.100326 1 *:142] activedefrag no
2025-07-08T13:29:05.100397466Z I0708 15:29:05.100332 1 *:142] lazyfree-lazy-server-del yes
2025-07-08T13:29:05.100399846Z I0708 15:29:05.100338 1 *:142] replica-read-only yes
2025-07-08T13:29:05.100402253Z I0708 15:29:05.100344 1 *:142] slave-read-only yes
2025-07-08T13:29:05.100404909Z I0708 15:29:05.100349 1 *:142] repl-diskless-load swapdb
2025-07-08T13:29:05.100407215Z I0708 15:29:05.100355 1 *:142] sanitize-dump-payload no
2025-07-08T13:29:05.100409438Z I0708 15:29:05.100360 1 *:142] lazyfree-lazy-expire yes
2025-07-08T13:29:05.100411662Z I0708 15:29:05.100367 1 *:142] debug-context ""
2025-07-08T13:29:05.100414038Z I0708 15:29:05.100373 1 *:142] repl-diskless-sync yes
2025-07-08T13:29:05.100416516Z I0708 15:29:05.100378 1 *:142] io-threads-do-reads no
2025-07-08T13:29:05.100422760Z I0708 15:29:05.100384 1 *:142] io-threads 1
2025-07-08T13:29:05.100425486Z I0708 15:29:05.100390 1 *:142] ------ FAST MEMORY TEST ------
2025-07-08T13:29:05.100462726Z W0708 15:29:05.100395 1 *:176] 19:S Bio worker thread #0 terminated
2025-07-08T13:29:05.100466133Z W0708 15:29:05.100424 1 *:176] 19:S Bio worker thread #1 terminated
2025-07-08T13:29:05.100468464Z W0708 15:29:05.100431 1 *:176] 19:S Bio worker thread #2 terminated
2025-07-08T13:29:05.100471463Z I0708 15:29:05.100437 1 *:142] *** Preparing to test memory region 77b000 (2293760 bytes)
2025-07-08T13:29:05.100473940Z I0708 15:29:05.100444 1 *:142] *** Preparing to test memory region 38bc6000 (2478080 bytes)
2025-07-08T13:29:05.100480115Z I0708 15:29:05.100450 1 *:142] *** Preparing to test memory region 7f489037f000 (201850880 bytes)
2025-07-08T13:29:05.100482795Z I0708 15:29:05.100456 1 *:142] *** Preparing to test memory region 7f489c400000 (8388608 bytes)
2025-07-08T13:29:05.100485278Z I0708 15:29:05.100461 1 *:142] *** Preparing to test memory region 7f489cc00000 (6291456 bytes)
2025-07-08T13:29:05.100487659Z I0708 15:29:05.100467 1 *:142] *** Preparing to test memory region 7f489d37e000 (8388608 bytes)
2025-07-08T13:29:05.100493138Z I0708 15:29:05.100472 1 *:142] *** Preparing to test memory region 7f489db7f000 (8388608 bytes)
2025-07-08T13:29:05.100498269Z I0708 15:29:05.100478 1 *:142] *** Preparing to test memory region 7f489e380000 (8388608 bytes)
2025-07-08T13:29:05.100504170Z I0708 15:29:05.100483 1 *:142] *** Preparing to test memory region 7f489eb80000 (11010048 bytes)
2025-07-08T13:29:05.100506830Z I0708 15:29:05.100489 1 *:142] *** Preparing to test memory region 7f489f75d000 (20480 bytes)
2025-07-08T13:29:05.100509198Z I0708 15:29:05.100494 1 *:142] *** Preparing to test memory region 7f489f98d000 (65536 bytes)
2025-07-08T13:29:05.100514643Z I0708 15:29:05.100500 1 *:142] *** Preparing to test memory region 7f489ff11000 (12288 bytes)
2025-07-08T13:29:05.100520303Z I0708 15:29:05.100506 1 *:142] *** Preparing to test memory region 7f48a00b6000 (8192 bytes)
2025-07-08T13:29:06.010405103Z I0708 15:29:06.010248 1 *:142] O.O.
2025-07-08T13:29:06.010695951Z I0708 15:29:06.010520 1 *:142] O.O.O
2025-07-08T13:29:06.010704225Z I0708 15:29:06.010537 1 *:142] Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
2025-07-08T13:29:06.010707554Z I0708 15:29:06.010609 1 *:142] === VALKEY BUG REPORT END. Make sure to include from START to END. ===
2025-07-08T13:29:06.010710975Z I0708 15:29:06.010614 1 *:142] Please report the crash by opening an issue on github:
2025-07-08T13:29:06.010713398Z I0708 15:29:06.010618 1 *:142] https://github.com/valkey-io/valkey/issues
2025-07-08T13:29:06.010716362Z I0708 15:29:06.010622 1 *:142] If a module was involved, please open in the module's repo instead.
2025-07-08T13:29:06.010718942Z I0708 15:29:06.010625 1 *:142] Suspect RAM error? Use valkey-server --test-memory to verify it.
2025-07-08T13:29:06.010721453Z I0708 15:29:06.010629 1 *:142] Some other issues could be detected by valkey-server --check-system
2025-07-08T13:29:06.223828518Z E0708 15:29:06.223641 1 *:345] Error during redis-server execution, errorsignal: aborted (core dumped)
2025-07-08T13:29:06.223852499Z I0708 15:29:06.223701 1 *:348] redis-server stopped
2025-07-08T13:29:06.223855867Z I0708 15:29:06.223709 1 *:38] Shutdown handler called
2025-07-08T13:29:06.223892364Z E0708 15:29:06.223823 1 *:313] http server stop error: http: Server closed
2025-07-08T13:29:06.280118253Z I0708 15:29:06.279893 1 *:321] Failover by address: 192.168.103.40:6380
2025-07-08T13:29:06.288618584Z I0708 15:29:06.288418 1 *:332] Target node was not a master, no failover necessary.
2025-07-08T13:29:06.288659791Z I0708 15:29:06.288460 1 *:726] Forgetting node with address: 192.168.103.40:6380
2025-07-08T13:29:06.298018317Z I0708 15:29:06.297811 1 *:574] forgetting node with ID: 5a874147b478800c3c5177e0a7c68f1d781977af
Additional information
OS Version: (SUSE Linux) 13.3.0
So far, this crashed occurred only once.
Taking a look. @eaarpms Are there any steps to reproduce it or just setting up cross version cluster (8.0 -> 8.1) causes the issue?
The crash happens around the clustering code flow.
While processing a MEET packet from an unknown sender, we create a temporary node and attach it to the inbound link in cluster_legacy.c:3228.
https://github.com/valkey-io/valkey/blob/f77cbfbe112423eeca380f141dae2c72624c6135/src/cluster_legacy.c#L3218-L3231
Note: The sender is NULL here because the node attached to link here is in HANDSHAKE state:
https://github.com/valkey-io/valkey/blob/f77cbfbe112423eeca380f141dae2c72624c6135/src/cluster_legacy.c#L3137
https://github.com/valkey-io/valkey/blob/f77cbfbe112423eeca380f141dae2c72624c6135/src/cluster_legacy.c#L2936-L2954
And I see a node in handshake state in the log above:
e7d3c787d77ce2764cd27c850fe1a24b8cd70f49 192.168.117.20:6380@16380,,tls-port=6380,shard-id=2b19f36b9329a4dcddfea4a391cd4b47835539a8 handshake - 0 0 0 disconnected
And crashes here:
https://github.com/valkey-io/valkey/blob/f77cbfbe112423eeca380f141dae2c72624c6135/src/cluster_legacy.c#L1388
What the crash indicates here is that the link seems to have a node already attached to it.
Looks like related to the change https://github.com/valkey-io/valkey/pull/1307, we can send MEET (multiple times) which doesn't seem to be handled in older versions.
We can backport #1441 / #1436 / #1307 to have stability between inter Valkey version but it will bite us with cross engine scenarios.
Tagging the authors / maintainers involved around the above changes: @enjoy-binbin / @pieturin / @madolson
Tried writing a test to reproduce this scenario, didn't manage to as of now. @enjoy-binbin Did you ever manage to create one while creating the temporary fix #1436 ?
I forgot the details, it is weird that i did not write a test (i normally do). I will think about the test when i full more time in the next week.
so it will happen when there is a handshake timeout during the downgrade?
I forgot the details, it is weird that i did not write a test (i normally do). I will think about the test when i full more time in the next week.
so it will happen when there is a handshake timeout during the downgrade?
That's my current hypothesis from all the information present in the logs. Tried explaining it here: https://github.com/valkey-io/valkey/issues/2341#issuecomment-3064181246
Hello @hpatro - we cannot reproduce it unfortunately, so far it happened only once.
BR/Marta
@eaarpms The crash happened in the replica running 8.0.3, right? This means that to prevent this crash we will need to backport a fix to 8.0.x?
You mentioned "this crashed occurred only once". Is this still true?
The assert failure cluster_legacy.c:1388 '!link->node' is not true in setClusterNodeToInboundClusterLink can be traced backwards. There are only two calls to this function. One of them is guarded by a check on !link->node. Thus the call leading up to this must be this one: https://github.com/valkey-io/valkey/blob/8.0.3/src/cluster_legacy.c#L3227
In 8.1, there is a commit that adds a check on !link->node on this code path: e4179f1f3b6b8d8490077860bd438303d302f2b9 from PR #1441. Can we backport this commit, or a subset of it, to 8.0.x?
There are some more PRs related to similar crashes and to re-sending of CLUSTER MEET, but all targeting 8.1 and later: #1436 also mentions the same/similar assert failure. Several PRs depend on each other here.
What's the minimal patch we pick to 8.0 to prevent this crash?
@pieturin @enjoy-binbin @hpatro
Hello @zuiderkwast
yes, the crash happened on 8.0.3 so it would be ok to backport to 8.0.x
Took a look at this again.
I think it will be difficult to backport subset of changes and maintain. Rather, I think we should avoid sending multiple meet packet to the nodes which doesn't support it (8.0 or less).
This is fixed by #2341 which is released in 8.1.5 and will be included in 9.0.1 too (to be released soon).
The newer node detects that the older node doesn't support a certain message (resend of CLUSTER MEET) so it will not send it in this case.