add index failed with error “Error 1105 (HY000): request pd http api failed with status: '503 Service Unavailable” when kill pd leader

[Lily2025]

Bug Report

What did you do?

1、add index 2、kill pd leader

What did you expect to see?

add index can success

What did you see instead?

add index failed with error “Error 1105 (HY000): request pd http api failed with status: '503 Service Unavailable”

What version of PD are you using (pd-server -V)?

./pd-server -V Release Version: v8.2.0-alpha Edition: Community Git Commit Hash: 1679dbca25b3483d1375c7e747da27e99ad77360 Git Branch: heads/refs/tags/v8.2.0-alpha UTC Build Time: 2024-04-30 11:39:01

[Lily2025]

/type bug /severity major /assign rleungx

[Lily2025]

/assign JmPotato

[Lily2025]

lightning failed with error “request pd http api failed with status: '500 Internal Server Error'“ when kill pd leader

lightning logs： [2024/05/06 15:42:15.072 +00:00] [ERROR] [client.go:234] ["[pd] request failed with a non-200 status"] [source=lightning] [name=GetStores] [url=http://tc-pd-2.tc-pd-peer.ha-test-lightning-tps-7575769-1-386.svc:2379/pd/api/v1/stores] [method=GET] [caller-id=pd-http-client] [status="500 Internal Server Error"] [body="[PD:apiutil:ErrRedirect]redirect failed\n"] [stack="github.com/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:234\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry.func1\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:139\ngithub.com/tikv/pd/client/retry.(*Backoffer).Exec\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/retry/backoff.go:78\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:160\ngithub.com/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:379\ngithub.com/tikv/pd/client/http.(*client).GetStores\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:403\ngithub.com/pingcap/tidb/pkg/lightning/tikv.ForAllStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/tikv/tikv.go:103\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*switcher).switchTiKVMode\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/local/tikv_mode.go:69\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*switcher).ToImportMode\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/local/tikv_mode.go:53\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*Controller).buildRunPeriodicActionAndCancelFunc.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/import.go:1001"] [2024/05/06 15:42:16.034 +00:00] [ERROR] [client.go:206] ["[pd] do http request failed"] [source=lightning] [name=SetRegionLabelRule] [url=http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7575769-1-386.svc:2379/pd/api/v1/config/region-label/rule] [method=POST] [caller-id=pd-http-client] [error="Post \"http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7575769-1-386.svc:2379/pd/api/v1/config/region-label/rule\": dial tcp 10.200.72.140:2379: connect: connection refused"] [stack="github.com/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:206\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry.func1\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:139\ngithub.com/tikv/pd/client/retry.(*Backoffer).Exec\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/retry/backoff.go:78\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:160\ngithub.com/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:379\ngithub.com/tikv/pd/client/http.(*client).SetRegionLabelRule\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:691\ngithub.com/pingcap/tidb/br/pkg/pdutil.pauseSchedulerByKeyRangeWithTTL\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:702\ngithub.com/pingcap/tidb/br/pkg/pdutil.PauseSchedulersByKeyRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:670\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).ImportEngine\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/local/local.go:1291\ngithub.com/pingcap/tidb/pkg/lightning/backend.(*ClosedEngine).Import\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/backend.go:373\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importKV\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/table_import.go:1346\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importEngine\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/table_import.go:920\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importEngines.func3\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/table_import.go:526"] [2024/05/06 15:42:16.060 +00:00] [ERROR] [backend.go:378] ["import failed"]

[JmPotato]

Introduced by #7896, multi-errors will return an error even if the request eventually succeeds after retries.