tiflow
tiflow copied to clipboard
pingcap
ticdc: panic: The CommitTs must be greater than the resolvedTs
What did you do?
- create a tidb changefeed
- downstream tidb cluster reload tikv tls crt
What did you expect to see?
No response
What did you see instead?
panic: The CommitTs must be greater than the resolvedTs
goroutine 237902 [running]: go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x0?, 0x4?, {0x4?, 0x0?, 0x0?}) go.uber.org/[email protected]/zapcore/entry.go:196 +0x78 go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x40032148f0, {0x406c378700, 0x4, 0x4}) go.uber.org/[email protected]/zapcore/entry.go:262 +0x2c0 go.uber.org/zap.(*Logger).Panic(0x406c378700?, {0x458ba13?, 0x406b4c17f8?}, {0x406c378700, 0x4, 0x4}) go.uber.org/[email protected]/logger.go:284 +0x54 github.com/pingcap/log.Panic({0x458ba13?, 0x36?}, {0x406c378700?, 0x0?, 0x4010e54800?}) github.com/pingcap/[email protected]/global.go:54 +0x94 github.com/pingcap/tiflow/cdc/kv.handleEventEntry(0x44f32cd?, 0x63173db91380027, 0x40175b82c0, 0x400411eb00, 0x406b4c1c28) github.com/pingcap/tiflow/cdc/kv/region_worker.go:677 +0xbd8 github.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).handleEventEntry(0x4034cfd000, {0x505f6a0?, 0x402efb5810?}, 0x4017a56028, 0x40175b82c0) github.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:171 +0x2e0 github.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).processEvent(0x4034cfd000, {0x505f6a0, 0x402efb5810}, {{0x4072ec02d0, 0x40175b82c0}, {0x0, {0x0, 0x0, 0x0}}, 0x403f5b5720, ...}) github.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:133 +0xf0 github.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).run(0x4034cfd000, {0x505f6a0, 0x402efb5810}) github.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:100 +0xd0 github.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func1() github.com/pingcap/tiflow/cdc/kv/shared_client.go:243 +0x2c golang.org/x/sync/errgroup.(*Group).Go.func1() golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 238148 golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x98
Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
(paste TiDB cluster version here)
Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)
TiCDC version (execute cdc version):
Release Version: v7.6.0-alpha
Git Commit Hash: 858906579a24c2d0768968a5020cf1db71411d45
Git Branch: heads/refs/tags/v7.6.0-alpha
UTC Build Time: 2023-12-11 11:44:25
Go Version: go version go1.21.5 linux/arm64
Failpoint Build: false
This issue is found once on the release-7.5 branch by running the following command in my local test environment:
tiup playground v7.5.1 --db 1 --kv 1 --pd 1 --tiflash 0 --without-monitor
./bin/cdc cli changefeed create --pd=http://127.0.0.1:2379/ --sink-uri="blackhole://" --changefeed-id="blackhole-test"
go-tpc tpcc -H [127.0.0.1](http://127.0.0.1/) -P 4000 -D workload --warehouses 20 -T 4 prepare
See https://github.com/pingcap/tiflow/issues/7672#issuecomment-1340313586
