karmada icon indicating copy to clipboard operation
karmada copied to clipboard

ci: etcd using the /livez for LivenessProbe and /readyz for StartupProbe.

Open liangyuanpeng opened this issue 1 year ago • 6 comments

What type of PR is this?

/kind cleanup etcd using the /livez for LivenessProbe and /readyz for StartupProbe at CI first, part of https://github.com/karmada-io/karmada/issues/5419 What this PR does / why we need it:

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

liangyuanpeng avatar Aug 27 '24 14:08 liangyuanpeng

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign xishanyongye-chang for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot avatar Aug 27 '24 14:08 karmada-bot

/cc @zhzhuang-zju

liangyuanpeng avatar Aug 27 '24 14:08 liangyuanpeng

:warning: Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 30.85%. Comparing base (a184b40) to head (d4ecff3).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5438      +/-   ##
==========================================
- Coverage   30.85%   30.85%   -0.01%     
==========================================
  Files         640      640              
  Lines       44384    44384              
==========================================
- Hits        13695    13694       -1     
- Misses      29709    29710       +1     
  Partials      980      980              
Flag Coverage Δ
unittests 30.85% <ø> (-0.01%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Aug 27 '24 14:08 codecov-commenter

/assign

zhzhuang-zju avatar Aug 28 '24 01:08 zhzhuang-zju

@liangyuanpeng The livenessProbe should properly reflect the fact that etcd needs a restart. Do you know how /livez accomplishes this? One benefit of using etcdctl get /registry --prefix --keys-only for the liveness check is that if it succeeds, then etcd's external functionality is normal. In addition, when I used /livez locally, I noticed that kubelet logs the following:

Aug 28 02:48:37 karmada-host-control-plane kubelet[682]: E0828 02:48:37.991001     682 prober.go:239] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=13477 actualBytes=10240

This can cause etcd to restart continuously.

zhzhuang-zju avatar Aug 28 '24 03:08 zhzhuang-zju

@zhzhuang-zju Thank you for your time! seems like this have some weird problem, let me make it to draft first.

Do you know how /livez accomplishes this

https://docs.google.com/document/d/1PaUAp76j1X92h3jZF47m32oVlR8Y-p-arB5XOB7Nb6U/edit#heading=h.78g8a1kutqd8 And also update it to issue.

liangyuanpeng avatar Aug 28 '24 13:08 liangyuanpeng