Add support for livenessProbe and ReadinessProbe of driver and executor pods

[giangnd1999]

[Meghana-Naik-Nice]

Any updates on this issue? We need this support as well As per the organization policy every pod needs livenessProbe and ReadinessProbe

[jacobsalway]

Functionality wise, this would be easy enough to add to the operator, but I'm curious what would you want to set the probes to?

[Meghana-Naik-Nice]

Set the probes to driver and executor pods which are launched after spark-submit

[jacobsalway]

@Meghana-Naik-Nice I mean what would you set the probe values to be? You might be able to use the Spark UI with a HTTP check on the driver but I'm not sure what you'd check on the executors

[yassan]

Sorry to jump in, but for example, if we want to run a Spark Connect server, we might need to check whether the gRPC service is functioning correctly.

In this case, a gRPC probe would be appropriate, as described in the Kubernetes documentation: Define a gRPC liveness probe.

[ChenYi015]

I think livenessProbe and readinessProbe can be configured by using the pod template feature, see pod template example here.