What is the behavior of HA database configurations wrt readiness probes?

[dmarkwat]

trafficstars

Question

We are configuring HA databases and the idea is that when one of them is down, the other should still be accessible -- bc HA, of course. Will readiness still report as operable if one of the replicas configured is down? What will happen in the event all are down? Will readiness still report OK?

The doc is mum on this which is why we pursued this, as we'd observed behavior that suggested when one was down readiness was no longer OK and the pods were pulled from their k8s Services. But looking at the code once more, this may have been isolated to cloud sql v1 as I'm not seeing how that would manifest here unless stopped or started were somehow tripping the failure.

From https://github.com/GoogleCloudPlatform/cloud-sql-proxy/pull/2419

Code

Additional Details

No response

[enocom]

@hessjcg @jackwotherspoon we should probably include guidance about using application-level readiness checks to determine the health of a database connection.

[jackwotherspoon]

Yes, this is a good idea, we should definitely add guidance on application readiness checks to check database connection health and not solely if Proxy is running.

[hessjcg]

I'm changing this to a docs issue. This is still a good idea that we should pursue.

[hessjcg]

Hi, @dmarkwat,

We added some documentation that describes the newer behavior of Proxy V2 readiness checks in kubernetes

In short, we would recommend you use "startup" and "liveness" healthchecks, but do not use a "readiness" health check on the proxy pod. Especially for the HA scenario that you have described.

I'll close this issue for now. Please reopen if you have further questions.