helm-charts icon indicating copy to clipboard operation
helm-charts copied to clipboard

Ethereum CL ReadinessProbe issue

Open J1a-wei opened this issue 2 years ago • 0 comments

Issue

Hi, I doubt that Is it feasible to use only ready probes, for example, the block is just behind (but not restarted. restart will cut the grpc conection).

In this scenario, the k8s readiness probe cannot cut off the grpc (I did an experiment). Then whether such readiness probe is still effective.

Why not use a liveness probe

I'm afraid it will cause an endless cycle. The CL is not synchronized well. The liveness probe restarts it. After restarting, it is still not synchronized well, entering an endless cycle. So I only used the readiness probe

Example

validator

  - args:
    - --beacon-rpc-provider=ethereum2-prater-beacon-01.svc.cluster.local:4000,ethereum2-prater-beacon-02.svc.cluster.local:4000

We have two CL service

  • ethereum2-prater-beacon-01.svc.cluster.local:4000
  • ethereum2-prater-beacon-02.svc.cluster.local:4000

When one CL or its corresponding EL falls behind (If there is no liveness probe, it will not restart), Kubelet does not cut this connect ( May be is keepalive? I did an experiment. It's true that grpc can't be cut. Unless you restart the upstream container) The validator will throw an error at this time, I'm not sure whether the validator can automatically transfer to the second service.

I appreciate any help!

Question

The k8s documentation states that the ready probe can abort traffic, but this is not feasible for grpc image

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes

J1a-wei avatar Dec 02 '22 11:12 J1a-wei