pika3.5.5 哨兵模式主从切换后从节点无法同步主节点数据

[yili1991]

Is this a regression?

Yes

Description

pika366哨兵模式有三个节点A、B、C，统一升级到355版本后，C为主节点，C挂掉后哨兵拉起B作为新主，之后BC之间正常同步数据，A无法从B同步数据

Please provide a link to a minimal reproduction of the bug

No response

Screenshots or videos

• 节点A info信息

• 节点B info信息

• 节点C info信息

• 节点A error log日志

• 节点B error log日志

• 节点C无error log

Please provide the version you discovered this bug in (check about page for version information)

Version:3.5.5

Anything else?

• 部分节点A的pika运行日志： path : ./conf/pika.conf -----------Pika server---------- pika_version: 3.5.5 pika_git_sha:0 pika_build_compile_date: 2024-09-20 08:41:17 redis_version: 3.5.5 WARNING: Logging before InitGoogleLogging() is written to STDERR I20240920 17:20:29.963819 1742395 base_conf.cc:336] 1 port 9221 I20240920 17:20:29.963891 1742395 base_conf.cc:336] 2 thread-num 1 I20240920 17:20:29.963897 1742395 base_conf.cc:336] 3 thread-pool-size 12 I20240920 17:20:29.963903 1742395 base_conf.cc:336] 4 sync-thread-num 6 I20240920 17:20:29.963908 1742395 base_conf.cc:336] 5 log-path ./log/ I20240920 17:20:29.963914 1742395 base_conf.cc:336] 6 db-path ./db/ I20240920 17:20:29.963920 1742395 base_conf.cc:336] 7 write-buffer-size 268435456 I20240920 17:20:29.963927 1742395 base_conf.cc:336] 8 arena-block-size 33554432 I20240920 17:20:29.963932 1742395 base_conf.cc:336] 9 timeout 60 I20240920 17:20:29.963937 1742395 base_conf.cc:336] 10 requirepass Dareway@2024 I20240920 17:20:29.963943 1742395 base_conf.cc:336] 11 masterauth Dareway@2024 I20240920 17:20:29.963948 1742395 base_conf.cc:336] 12 userpass Dareway@2024 I20240920 17:20:29.963954 1742395 base_conf.cc:336] 13 userblacklist I20240920 17:20:29.963960 1742395 base_conf.cc:336] 14 instance-mode classic I20240920 17:20:29.963965 1742395 base_conf.cc:336] 15 databases 1 I20240920 17:20:29.963971 1742395 base_conf.cc:336] 16 default-slot-num 1024 I20240920 17:20:29.963976 1742395 base_conf.cc:336] 17 replication-num 0 I20240920 17:20:29.963982 1742395 base_conf.cc:336] 18 consensus-level 0 I20240920 17:20:29.963989 1742395 base_conf.cc:336] 19 dump-prefix I20240920 17:20:29.963994 1742395 base_conf.cc:336] 20 dump-path ./dump/ I20240920 17:20:29.963999 1742395 base_conf.cc:336] 21 dump-expire 0 I20240920 17:20:29.964005 1742395 base_conf.cc:336] 22 pidfile ./pika.pid I20240920 17:20:29.964010 1742395 base_conf.cc:336] 23 maxclients 20000 I20240920 17:20:29.964016 1742395 base_conf.cc:336] 24 target-file-size-base 20971520 I20240920 17:20:29.964021 1742395 base_conf.cc:336] 25 expire-logs-days 7 I20240920 17:20:29.964027 1742395 base_conf.cc:336] 26 expire-logs-nums 10 I20240920 17:20:29.964033 1742395 base_conf.cc:336] 27 root-connection-num 2 I20240920 17:20:29.964038 1742395 base_conf.cc:336] 28 slowlog-write-errorlog no I20240920 17:20:29.964044 1742395 base_conf.cc:336] 29 slowlog-log-slower-than 10000 I20240920 17:20:29.964051 1742395 base_conf.cc:336] 30 slowlog-max-len 128 I20240920 17:20:29.964056 1742395 base_conf.cc:336] 31 db-sync-path ./dbsync/ I20240920 17:20:29.964061 1742395 base_conf.cc:336] 32 db-sync-speed 1024 I20240920 17:20:29.964066 1742395 base_conf.cc:336] 33 slave-priority 100 I20240920 17:20:29.964072 1742395 base_conf.cc:336] 34 sync-window-size 9000 I20240920 17:20:29.964078 1742395 base_conf.cc:336] 35 max-conn-rbuf-size 268435456 I20240920 17:20:29.964083 1742395 base_conf.cc:336] 36 write-binlog yes I20240920 17:20:29.964089 1742395 base_conf.cc:336] 37 binlog-file-size 104857600 I20240920 17:20:29.964095 1742395 base_conf.cc:336] 38 max-cache-statistic-keys 0 I20240920 17:20:29.964100 1742395 base_conf.cc:336] 39 small-compaction-threshold 5000 I20240920 17:20:29.964107 1742395 base_conf.cc:336] 40 max-write-buffer-size 10737418240 I20240920 17:20:29.964111 1742395 base_conf.cc:336] 41 max-write-buffer-number 2 I20240920 17:20:29.964118 1742395 base_conf.cc:336] 42 max-client-response-size 1073741824 I20240920 17:20:29.964123 1742395 base_conf.cc:336] 43 compression snappy I20240920 17:20:29.964128 1742395 base_conf.cc:336] 44 max-background-flushes 1 I20240920 17:20:29.964134 1742395 base_conf.cc:336] 45 max-background-compactions 2 I20240920 17:20:29.964140 1742395 base_conf.cc:336] 46 max-cache-files 5000 I20240920 17:20:29.964145 1742395 base_conf.cc:336] 47 max-bytes-for-level-multiplier 10 I20240920 17:20:29.964151 1742395 base_conf.cc:336] 48 slaveof B:9331 I20240920 17:20:29.964156 1742395 base_conf.cc:336] 49 admin-cmd-list info, monitor, ping I20240920 17:20:29.964162 1742395 base_conf.cc:336] 50 replication-id ac78114eef217ce1037ccdc77783fd49e6a3bd0bde0931c3d0 I20240920 17:20:29.964193 1742395 base_conf.cc:336] 51 run-id c18e33d335cc558d4a724a3ccf8f8ff6a315252d I20240920 17:20:29.964200 1742395 base_conf.cc:336] 52 internal-used-unfinished-full-sync I20240920 17:20:29.964219 1742395 pika.cc:209] Server at: ./conf/pika.conf I20240920 17:20:29.964485 1742395 net_interfaces.cc:104] Using Networker Interface: eno145 I20240920 17:20:29.966964 1742395 net_interfaces.cc:148] got ip A I20240920 17:20:29.966989 1742395 pika_server.cc:156] host: A port: 9221 I20240920 17:20:29.967003 1742395 pika_server.cc:70] Worker queue limit is 20100 W20240920 17:20:29.967011 1742395 pika_server.cc:71] 0.0.0.0 I20240920 17:20:29.967927 1742395 pika_server.cc:1665] Dump file is not exist,path: ./dump/ I20240920 17:20:29.968278 1742395 pika_binlog.cc:98] Binlog: Find the exist file. W20240920 17:20:29.968418 1742395 pika_binlog_reader.cc:207] Read BadRecord record, will decode failed, this record may dbsync padded record, not processed here E20240920 17:20:29.968431 1742395 pika_binlog_transverter.cc:167] Binlog Item type error, expect type:1 actualy type: 12032 W20240920 17:20:29.968492 1742395 pika_stable_log.cc:187] Binlog item decode failed I20240920 17:20:30.127966 1742395 pika_db.cc:49] db0 DB Success I20240920 17:20:30.128170 1742583 pika_cache_load_thread.cc:186] PikaCacheLoadThread::ThreadMain Start I20240920 17:20:30.130649 1742395 net_util.cc:121] TimerTaskThread Starting... I20240920 17:20:30.131000 1742395 pika_server.cc:213] Pika Server going to start I20240920 17:20:30.131016 1742395 rsync_server.cc:48] start RsyncServer ... I20240920 17:20:30.131359 1742395 rsync_server.cc:60] RsyncServer started ... I20240920 17:20:30.131649 1742614 pika_repl_client.cc:199] Try Send Meta Sync Request to Master (B:9331) W20240920 17:20:30.132860 1742585 pika_repl_client_conn.cc:136] Meta Sync Failed: replicationid on both sides of the connection are inconsistent W20240920 17:20:30.132918 1742585 pika_server.cc:684] Sync error, set repl_state to PIKA_REPL_ERROR I20240920 17:20:30.132951 1742584 client_thread.cc:359] received kNotiClose I20240920 17:20:30.133020 1742584 pika_repl_client_thread.cc:20] ReplClient Close conn, fd=78, ip_port=B:11331 I20240920 17:20:30.534910 1742612 dispatch_thread.cc:151] accept new conn 75:C:48458:0 I20240920 17:20:30.534950 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.534984 1742612 dispatch_thread.cc:151] accept new conn 76:C:48460:0 I20240920 17:20:30.535092 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.543035 1742612 dispatch_thread.cc:151] accept new conn 77:A:45752:0 I20240920 17:20:30.543071 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.543100 1742612 dispatch_thread.cc:151] accept new conn 78:A:45754:0 I20240920 17:20:30.543112 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.543141 1742612 dispatch_thread.cc:151] accept new conn 79:A:37028:0 I20240920 17:20:30.543151 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.543171 1742612 dispatch_thread.cc:151] accept new conn 80:A:37030:0 I20240920 17:20:30.543181 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.756268 1742612 dispatch_thread.cc:151] accept new conn 81:B:52700:0 I20240920 17:20:30.756398 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:30.756438 1742612 dispatch_thread.cc:151] accept new conn 82:B:52702:0 I20240920 17:20:30.756458 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:39.646625 1742612 dispatch_thread.cc:151] accept new conn 83:A:45822:0 I20240920 17:20:39.646687 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:20:56.832079 1742608 pika_command.cc:1061] command: info keyspace 1 I20240920 17:21:05.428328 1742609 pika_command.cc:1061] command: info keyspace 1 I20240920 17:21:08.944960 1742608 pika_command.cc:1061] command: info keyspace 1 I20240920 17:22:18.149905 1742610 worker_thread.cc:262] connection fd: 83, ip_port: A:45822, name: , is_reply: 0, close: 0 keepalive timeout, the keepalive_timeout_ is 60 I20240920 17:22:32.365897 1742612 dispatch_thread.cc:151] accept new conn 83:A:46476:0 I20240920 17:22:32.365960 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:22:45.480271 1742608 pika_command.cc:1061] command: info keyspace 1 I20240920 17:22:48.168867 1742609 pika_command.cc:1061] command: info keyspace 1 I20240920 17:24:12.161893 1742610 worker_thread.cc:262] connection fd: 83, ip_port: A:46476, name: , is_reply: 0, close: 0 keepalive timeout, the keepalive_timeout_ is 60 I20240920 17:28:05.465426 1742612 dispatch_thread.cc:151] accept new conn 83:A:47582:0 I20240920 17:28:05.465494 1742612 dispatch_thread.cc:159] find worker(0), refresh the last_thread_ to 0 I20240920 17:28:17.772475 1742604 pika_server.cc:697] Remove Master Success, ip_port: B:9331 I20240920 17:28:32.256162 1742603 base_conf.cc:345] ret OK I20240920 17:28:32.256547 1752638 pika_server.cc:1798] clear cache start... I20240920 17:28:32.256592 1742599 base_conf.cc:345] ret OK I20240920 17:28:32.256605 1752638 pika_server.cc:1802] clear cache finish

• 部分节点A的pika运行日志2： 1445394:X 23 Sep 2024 16:44:05.087 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:45:36.006 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:46:06.024 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:46:36.034 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:47:06.105 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:47:32.467 # +sdown master mymaster1 B 9331 1445394:X 23 Sep 2024 16:47:32.598 * Sentinel new configuration saved on disk 1445394:X 23 Sep 2024 16:47:32.598 # +new-epoch 4 1445394:X 23 Sep 2024 16:47:32.607 * Sentinel new configuration saved on disk 1445394:X 23 Sep 2024 16:47:32.607 # +vote-for-leader 461094c8c4e9f902d556fef115b45051fb816393 4 1445394:X 23 Sep 2024 16:47:33.579 # +odown master mymaster1 B 9331 #quorum 2/2 1445394:X 23 Sep 2024 16:47:33.579 # Next failover delay: I will not start a failover before Mon Sep 23 16:53:33 2024 1445394:X 23 Sep 2024 16:47:36.078 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:47:39.435 # +config-update-from sentinel 461094c8c4e9f902d556fef115b45051fb816393 A 26380 @ mymaster1 B 9331 1445394:X 23 Sep 2024 16:47:39.435 # +switch-master mymaster1 B 9331 C 9221 1445394:X 23 Sep 2024 16:47:39.435 * +slave slave A:9221 A 9221 @ mymaster1 C 9221 1445394:X 23 Sep 2024 16:47:39.435 * +slave slave B:9331 B 9331 @ mymaster1 C 9221 1445394:X 23 Sep 2024 16:47:39.437 * Sentinel new configuration saved on disk 1445394:X 23 Sep 2024 16:48:06.080 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:48:09.459 # +sdown slave B:9331 B 9331 @ mymaster1 C 9221 1445394:X 23 Sep 2024 16:48:36.056 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:49:06.084 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:49:36.072 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:50:06.106 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:51:37.034 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:52:07.058 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:52:37.060 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:53:07.105 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:53:37.096 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:54:07.106 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:54:37.054 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:55:07.089 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:56:38.042 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:57:08.081 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:57:38.045 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:58:08.075 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:58:38.026 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 16:59:08.071 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 16:59:38.012 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 17:00:08.027 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 17:01:39.056 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 17:02:09.084 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 17:02:39.043 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 17:03:09.045 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 17:03:39.073 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 17:04:09.089 # -tilt #tilt mode exited 1445394:X 23 Sep 2024 17:04:39.039 # +tilt #tilt mode entered 1445394:X 23 Sep 2024 17:05:09.067 # -tilt #tilt mode exited

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

Title: pika3.5.5 The slave node cannot synchronize the master node data after the master-slave switch in sentry mode.

[Mixficsol]

后续看一下情况

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

Let’s take a look at the situation later