temporalio
Check DB Connectivity in Liveness probe
Is your feature request related to a problem? Please describe.
In K8s environment there are situations when DB passwords and certificates are renewed.
When that happens the current worker, frontend, matching and history behavior is to fail silently and log the error.
The passwords are stored in K8s secrets and loaded as environment variables, in other words, a pod restart would resolve the issue.
Describe the solution you'd like Check DB connectivity in liveness probe. If the DB password is changed the temporal pods would restart and load the new secret.
Describe alternatives you've considered
Using tctl namespace list as liveness probe exec command, but it can't connect to the localhost even if the correct port is specified.
Plus, replacing the existing liveness probe feels hacky.
Additional context
An example of the tctl n l failure when run inside the history pod.
temporal-history-67f566466-qz8n8:/etc/temporal$ tctl --address localhost:7234 n l
Error: Error when list namespaces info
Error Details: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
