CDC kvclient panic: runtime error: invalid memory address or nil pointer dereferenc

[fubinzh]

What did you do?

1. Deploy TiDB cluster with 6 TiKV and 2 CDC
2. Stop GC
3. Create blackhole changefeed and pause it
4. Run workload for several days to generate hube incremetal data for cdc t sync
5. Resume cdc changefeed to do initial scan

What did you expect to see?

CDC should not panic

What did you see instead?

CDC panic seen.

[WARN] TiCDC server data-dir is not set. Please use `cdc server --data-dir` to start the cdc server if possible.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x2c469fb]

goroutine 46056435 [running]:
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).enqueueError(0xc0014dcb00, {0x0, 0x0}, {{{0xc67c1, 0x23, 0xd5b}, {{0xc0fe558120, 0x2d, 0x30}, {0xc0fe558150, ...}}, ...}, ...})
        github.com/pingcap/tiflow/cdc/kv/client.go:870 +0x9b
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).onRegionFail(0xc0014dcb00, {0x0, 0x0}, {{{0xc67c1, 0x23, 0xd5b}, {{0xc0fe558120, 0x2d, 0x30}, {0xc0fe558150, ...}}, ...}, ...})
        github.com/pingcap/tiflow/cdc/kv/client.go:555 +0xf0
github.com/pingcap/tiflow/cdc/kv.(*regionWorker).evictAllRegions(0xc0ce863440)
        github.com/pingcap/tiflow/cdc/kv/region_worker.go:824 +0x298
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).receiveFromStream(0xc0014dcb00, {0x42f9b68?, 0xc003a79940?}, 0xc0c2ab3f38?, {0xc0009cfdc0, 0x3d}, 0x5, {0x4319a38, 0xc0cfd50230}, 0xc02cd5a100)
        github.com/pingcap/tiflow/cdc/kv/client.go:1067 +0x13fd
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).requestRegionToStore.func2()
        github.com/pingcap/tiflow/cdc/kv/client.go:657 +0xb2
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
        golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

/ # /tikv-server -V
TiKV
Release Version:   6.5.3
Edition:           Community
Git Commit Hash:   0578b41fca52aeee5ea5708efd41275165242988
Git Commit Branch: heads/refs/tags/v6.5.3-pr16054-0578b4
UTC Build Time:    2023-11-23 08:19:34
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

TiCDC version (execute cdc version):

bash-5.1# /cdc version
Release Version: v6.5.3-20231116-255d810
Git Commit Hash: 255d8104c8b9b4f8af191ee69d301ced723ccc3d
Git Branch: heads/refs/tags/v6.5.3-20231116-255d810
UTC Build Time: 2023-11-16 06:35:28
Go Version: go version go1.19.12 linux/amd64
Failpoint Build: false

[fubinzh]

/severity major

[asddongmen]

Does not affect v8.1.0 since the related code is outdate and removed. cc @flowbehappy