gravity-site has readiness probe failed if it not leader and produces thousands of events

[aocheretnoy-zz]

Describe the bug

kubectl describe pods -n=kube-system gravity-site-xxxxxx in case if the pod is not leader, produces events

Events:
  Type     Reason     Age                      From                                 Message
  ----     ------     ----                     ----                                 -------
  Warning  Unhealthy  86s (x25710 over 2d23h)  kubelet, ip.ec2.internal  Readiness probe failed: HTTP probe failed with statuscode: 503

To Reproduce

Execute for not leader kubectl describe pods -n=kube-system gravity-site-xxxxxx Expected behavior

pods should be healthy

Logs

Environment (please complete the following information):

• OS [e.g. Redhat 7.4]:
• Gravity [e.g. 5.5.4]:gravity-site:6.3.9
• Platform [e.g. Vmware, AWS]:

Additional context

[r0mant]

Thanks for the report! However, this is by design. In an HA cluster only a single gravity-site is ready and serves API requests. Going to close for now.

[r0mant]

Going to reopen actually for tracking. Not sure we're gonna be able to do something about it in near future as the internals rely quite heavily on this "single leader" model at the moment, but this is a valid complaint nonetheless.

As an idea, maybe we could achieve the same effect by utilizing pod labels and service selector instead of failing the readiness probe.