pd icon indicating copy to clipboard operation
pd copied to clipboard

lease: keep alive Leadership immediately after finish the election of leader

Open HuSharp opened this issue 1 year ago • 3 comments

What problem does this PR solve?

Issue Number: Ref #7251

What is changed and how does it work?

fix `etcdserver: requested lease not found`
// granted 3s
[2024/01/15 02:55:44.603 +00:00] [INFO] [server.go:1657] ["start to campaign PD leader"] [campaign-leader-name=tc-pd-0]
[2024/01/15 02:55:44.604 +00:00] [INFO] [lease.go:66] ["lease granted"] [lease-id=6177967866903092454] [lease-timeout=3] [purpose="leader election"]
// commit too slow
[2024/01/15 02:55:45.092 +00:00] [WARN] [etcdutil.go:152] ["kv gets too slow"] [request-key=/pd/7324144488088872732/member/5619078945150009647/leader_priority] [cost=2.764836133s] []
[2024/01/15 02:55:50.563 +00:00] [INFO] [etcdutil.go:314] ["update endpoints"] [num-change=3->2] [last-endpoints="[http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-1.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"] [endpoints="[http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"]
[2024/01/15 02:55:50.564 +00:00] [INFO] [etcdutil.go:314] ["update endpoints"] [num-change=3->2] [last-endpoints="[http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-1.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"] [endpoints="[http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"]
[2024/01/15 02:55:51.020 +00:00] [WARN] [etcd_kv.go:174] ["txn runs too slow"] [response="{\"header\":{\"cluster_id\":10740517942856960270,\"member_id\":17574458819109638910,\"revision\":1502,\"raft_term\":16},\"succeeded\":true,\"responses\":[{\"Response\":{\"ResponsePut\":{\"header\":{\"revision\":1502}}}}]}"] [cost=6.415850764s] []
// `02:55:51` pass 7s than granted
[2024/01/15 02:55:51.020 +00:00] [INFO] [leadership.go:181] ["check campaign resp"] [resp="{\"header\":{\"cluster_id\":10740517942856960270,\"member_id\":17574458819109638910,\"revision\":1502,\"raft_term\":16},\"succeeded\":true,\"responses\":[{\"Response\":{\"ResponsePut\":{\"header\":{\"revision\":1502}}}}]}"]
[2024/01/15 02:55:51.020 +00:00] [INFO] [leadership.go:190] ["write leaderData to leaderPath ok"] [leader-key=/pd/7324144488088872732/leader] [purpose="leader election"]
[2024/01/15 02:55:51.020 +00:00] [INFO] [server.go:1683] ["campaign PD leader ok"] [campaign-leader-name=tc-pd-0]
[2024/01/15 02:55:51.020 +00:00] [INFO] [server.go:1691] ["initializing the global TSO allocator"]
[2024/01/15 02:55:51.020 +00:00] [INFO] [tso.go:160] ["start to sync timestamp"] []
// check lease failed
[2024/01/15 02:55:51.020 +00:00] [INFO] [lease.go:155] ["start lease keep alive worker"] [interval=1s] [purpose="leader election"]
[2024/01/15 02:55:51.020 +00:00] [WARN] [lease.go:170] ["lease keep alive failed"] [purpose="leader election"] [start=2024/01/15 02:55:51.020 +00:00] [error="etcdserver: requested lease not found"]
...
[2024/01/15 02:55:51.038 +00:00] [INFO] [lease.go:187] ["stop lease keep alive worker"] [purpose="leader election"]
[2024/01/15 02:55:51.039 +00:00] [INFO] [server.go:1657] ["start to campaign PD leader"] [campaign-leader-name=tc-pd-0]
...

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

Release note

None.

HuSharp avatar Jan 15 '24 07:01 HuSharp

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment. After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review. Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot[bot] avatar Jan 15 '24 07:01 ti-chi-bot[bot]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

ti-chi-bot[bot] avatar Jan 15 '24 07:01 ti-chi-bot[bot]

Codecov Report

Merging #7707 (4292223) into master (cd0ffba) will decrease coverage by 0.18%. Report is 1 commits behind head on master. The diff coverage is 96.66%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7707      +/-   ##
==========================================
- Coverage   73.69%   73.51%   -0.18%     
==========================================
  Files         432      432              
  Lines       47805    47787      -18     
==========================================
- Hits        35229    35132      -97     
- Misses       9572     9637      +65     
- Partials     3004     3018      +14     
Flag Coverage Δ
unittests 73.51% <96.66%> (-0.18%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

codecov[bot] avatar Jan 15 '24 07:01 codecov[bot]

Do we still need this pr? Will it need to be picked to 7.5?

lhy1024 avatar May 20 '24 04:05 lhy1024

Do we still need this pr? Will it need to be picked to 7.5?

I don't think this pr is needed for now, Let's close it for now.

HuSharp avatar May 20 '24 05:05 HuSharp