secondary cluster cdc panic

[Lily2025]

What did you do?

1、restore data for primary and secondary 2、create changefeed and set bdr role for primary and secondary 3、run sysbench on primary and secondary 4、add index and then drop index on primary 5、kill one of tikv on primary

What did you expect to see?

no panic

What did you see instead?

after sometime when kill one of tikv on primary，secondary cluster cdc panic Explore-logs-2024-10-18 17_43_12.txt 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915681716+08:00 stderr F \tgolang.org/x/[email protected]/errgroup/errgroup.go:78 +0x50 fp=0xc006f21fe0 sp=0xc006f21f78 pc=0x1ac9f50","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.91567794+08:00 stderr F golang.org/x/sync/errgroup.(*Group).Go.func1()","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915670897+08:00 stderr F \tgithub.com/pingcap/tiflow/cdc/kv/shared_stream.go:213 +0x27 fp=0xc006f21f78 sp=0xc006f21f40 pc=0x3d0ca67","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915666246+08:00 stderr F github.com/pingcap/tiflow/cdc/kv.(*requestedStream).run.func5()","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915662339+08:00 stderr F \tgithub.com/pingcap/tiflow/cdc/kv/shared_stream.go:379 +0x1729 fp=0xc006f21f40 sp=0xc006f218f0 pc=0x3d0f0e9","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.91565457+08:00 stderr F github.com/pingcap/tiflow/cdc/kv.(*requestedStream).send(0xc004ceab40, {0x5e62ab0, 0xc003895810}, 0xc0071dc000, 0xc0067286c0)","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915650163+08:00 stderr F \tgithub.com/pingcap/tiflow/cdc/kv/shared_stream.go:262 +0x19f fp=0xc006f218f0 sp=0xc006f21750 pc=0x3d0f8df","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915645961+08:00 stderr F github.com/pingcap/tiflow/cdc/kv.(*requestedStream).send.func1()","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915641861+08:00 stderr F \truntime/select.go:335 +0x7a5 fp=0xc006f21750 sp=0xc006f21628 pc=0x4516a5","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915637366+08:00 stderr F runtime.selectgo(0xc006f21830, 0xc006f21784, 0x0?, 0x0, 0xc006f217d8?, 0x1)","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915630131+08:00 stderr F \truntime/proc.go:424 +0xce fp=0xc006f21628 sp=0xc006f21608 pc=0x47604e","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915625382+08:00 stderr F runtime.gopark(0xc006f21830?, 0x3?, 0x70?, 0x16?, 0xc006f2178a?)","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915607012+08:00 stderr F goroutine 12359 gp=0xc00432e700 m=nil [select]:","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915602936+08:00 stderr F ","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915599237+08:00 stderr F \tgolang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915595505+08:00 stderr F created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 12333","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915591377+08:00 stderr F \truntime/asm_amd64.s:1700 +0x1 fp=0xc00b571fe8 sp=0xc00b571fe0 pc=0x47edc1","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915583169+08:00 stderr F runtime.goexit({})","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915578833+08:00 stderr F \tgolang.org/x/[email protected]/errgroup/errgroup.go:78 +0x50 fp=0xc00b571fe0 sp=0xc00b571f78 pc=0x1ac9f50","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915575155+08:00 stderr F golang.org/x/sync/errgroup.(*Group).Go.func1()","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.915570698+08:00 stderr F \tgithub.com/pingcap/tiflow/cdc/kv/shared_stream.go:189 +0x2e fp=0xc00b571f78 sp=0xc00b571f30 pc=0x3d0ccae","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.91556707+08:00 stderr F github.com/pingcap/tiflow/cdc/kv.(*requestedStream).run.func3()","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.9155629+08:00 stderr F \tgithub.com/pingcap/tiflow/cdc/kv/shared_stream.go:227 +0x75 fp=0xc00b571f30 sp=0xc00b571cb0 pc=0x3d0d255","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.91555365+08:00 stderr F github.com/pingcap/tiflow/cdc/kv.(*requestedStream).receive(0xc004ceab40, {0x5e62ab0, 0xc003895810}, 0xc0071dc000, 0xc0067286c0, 0x0?, 0x0)","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20 {"pod":"cdc-downstream-ticdc-0","container":"ticdc","log":"2024-10-17T14:44:56.91554985+08:00 stderr F \tgithub.com/pingcap/[email protected]/pkg/cdcpb/cdcpb.pb.go:1683 +0x46 fp=0xc00b571cb0 sp=0xc00b571c80 pc=0x3cc4846","namespace":"endless-ha-test-bdr-ddl-tps-7635680-1-855"} 2024-10-17 14:45:20

Versions of the cluster

./cdc version Release Version: v8.4.0-alpha-40-g62d07b55e Git Commit Hash: 62d07b55e71d0f217886624859bd40eccfab04c8 Git Branch: HEAD UTC Build Time: 2024-10-16 17:13:21 Go Version: go1.23.2 Failpoint Build: false 2024-10-17T10:07:31.445+0800

current status of DM cluster (execute query-status <task-name> in dmctl)

No response