openwhisk-deploy-kube icon indicating copy to clipboard operation
openwhisk-deploy-kube copied to clipboard

Missing liveness and readiness probes in all components

Open larandersson opened this issue 6 years ago • 2 comments

It seems as if the livenessProbe and readinessProbe is missing for several components:

The controller is missing the readinessProbe, which causes issues with rolling updates. With 1 replica, the observed behaviour is that the controller becomes unavailable for a long time since Kubernetes has no way of knowing when the new pod is ready, thereby killing off the predesessor while the new one is still in init.

All other components are missing both the readinessProbe and the livenessProbe, so similar problems with rolling updates could be an issue with thos. Since it's a good practice to have these probes, they should be added.

larandersson avatar Feb 22 '19 08:02 larandersson

Agreed, but it is not always clear how to define a valid readiness probe.

I added them for zookeeper in #430.

There is some logic in https://github.com/apache/incubator-openwhisk-deploy-kube/blob/master/helm/openwhisk/templates/_readiness.tpl that may be able to be adapted for a few other pods.

I would love to find out a valid way to probe kafka for readiness (vs. liveness) externally so we could get rid of the kludge here: https://github.com/apache/incubator-openwhisk-deploy-kube/blob/115debb691ad228dccc9c0a9e6c32736f3319268/helm/openwhisk/templates/_readiness.tpl#L24-L25

dgrove-oss avatar Feb 22 '19 21:02 dgrove-oss

We should add one for Invoker as well as it support /ping endpoint. So having a config similar to one in Controller should be fine

chetanmeh avatar Sep 05 '19 05:09 chetanmeh