cdc changefeed stucks when injecting all tikv network partition

[fubinzh]

What did you do?

1. create mysql sink changefeed
2. run tpcc prepare

go-tpc tpcc --db workload --warehouses 80 -T 50 --host upstream-tidb.cdc-testbed
-tps-7597190-1-758 --port 4000 --parts 1 prepare --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062

3. run tpcc workload

/go-tpc tpcc --db workload --warehouses 80 -T 50 --host upstream-tidb.cdc-testbed
-tps-7597190-1-758 --port 4000 --parts 1 --time 1h0m0s run --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062'

4. Inject all tikv network loss periodically (inject for 10s every 5 minutes), tikv network loss injected during: 2024-07-02T10:26:36 - 10:26:46 2024-07-02T10:31:36 - 10:31:46 2024-07-02T10:36:36 - 10:36:46 2024-07-02T10:41:36 - 10:41:46 2024-07-02T10:46:36 - 10:46:46
5. check cdc changefeed status

What did you expect to see?

changefeed checkpoint should be normal

What did you see instead?

checkpoint and resolved ts stucks.

Versions of the cluster

[root@upstream-ticdc-0 log]# /cdc version Release Version: v8.2.0-alpha Git Commit Hash: 2db092244d18c043f7f8ecc85a598680b33cc324 Git Branch: heads/refs/tags/v8.2.0-alpha UTC Build Time: 2024-06-27 11:38:15 Go Version: go version go1.21.10 linux/amd64 Failpoint Build: false

[root@upstream-tikv-0 /]# /tikv-server -V TiKV Release Version: 8.2.0-alpha Edition: Community Git Commit Hash: e278928817ed4007279f90c7bf8e42c9b02213f2 Git Commit Branch: heads/refs/tags/v8.2.0-alpha UTC Build Time: 2024-07-01 11:38:18 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release