app: Implement a separate health check server
Do not merge before issue/resolution is accepted here: linkerd/linkerd2/issues/7560
A separate server provides identically behaving /live and /ready routes to the admin server. Does not remove the existing admin server's routes.
Background:
On some Kubernetes distributions, requests from the control plane may not come from a private address range IP address or even a consistent IP address. This poses a problem, because the admin server used in a multicluster mesh needs to simultaneously serve /live and /ready routes to:
- The Kubernetes control plane, for liveness and readiness probes respectively
- Remote clusters as part of probing for remote gateway
In order to avoid exposing the other admin routes, the multicluster gateway uses an authorization policy forbidding unauthorized and out-of-cluster requests. This causes the gateway to fail readiness and liveness probes.
Resolution:
Implement a separate server in the proxy app that can securely serve /live and /ready routes. The port that server listens on can be used for health check probes internally, without an authorization policy.
See: https://github.com/linkerd/linkerd2/pull/7548