pd icon indicating copy to clipboard operation
pd copied to clipboard

PD leader is abnormal when rolling restart

Open CharlesCheung96 opened this issue 11 months ago • 0 comments

Bug Report

What did you do?

  1. Deploy tidb cluster (with ticdc node) using operator.
  2. Rolling restart pd via configuration update.

What did you expect to see?

the cluster is normal

What did you see instead?

[2024/03/17 03:44:29.073 +00:00] [INFO] [server.go:1813] [\"no longer a leader because lease has expired, pd leader will step down\"]
  1. The session between PD and TiCDC is disconnected
[2024/03/17 03:44:31.716 +00:00] [WARN] [capture.go:604] ["session is disconnected"] [error="[CDC:ErrEtcdSessionDone]the etcd session is done"] [errorVerbose="[CDC:ErrEtcdSessionDone]the etcd session is done\[ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/[email protected]/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:190\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:595\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).campaignOwner.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:531\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650](http://ngithub.com/pingcap/errors.AddStack/n/tgithub.com/pingcap/[email protected]/errors.go:174/ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.com/pingcap/[email protected]/normalize.go:164/ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run/n/tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:190/ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker/n/tgithub.com/pingcap/tiflow/cdc/capture/capture.go:595/ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).campaignOwner.func4/n/tgithub.com/pingcap/tiflow/cdc/capture/capture.go:531/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/tgolang.org/x/[email protected]/errgroup/errgroup.go:78/nruntime.goexit/n/truntime/asm_amd64.s:1650)"]
  1. The session between PD and TiDB is disconnected
[2024/03/17 03:44:33.859 +00:00] [INFO] [manager.go:267] [\"etcd session is done, creates a new one\"] [\"owner info\"=\"[autoid] tidb/autoid/leader ownerManager upstream-tidb-1.upstream-tidb-peer.cdc-airbnb-simple-tian-gcp-tps-7500530-1-808.svc:10080\"]"}

What version of PD are you using (pd-server -V)?

6ee25f9f8177067957b0932f1be7e2cace0202d0

CharlesCheung96 avatar Mar 22 '24 07:03 CharlesCheung96