[Self-Hosted] Expand preflight checks and support bundle diagnostics for certificate misconfiguration

[adrienthebo]

Certificate issues are a significant source of friction in the Gitpod self-hosted installation experience; misconfiguration causes a variety of issues including breaking the installation of Gitpod and preventing prebuilds and workspaces from starting. Smoothing this process and providing diagnostics in case of failures will improve the overall Gitpod user experience.

Potential points of misconfiguration include (but are not limited to) the following:

• The certificate presented by an external container registry
• The certificate presented by the external VCS integration server
• Gitpod's public facing certificate (certificate/https-certificates)

Gitpod's public facing certificate is currently the highest friction point. As Gitpod supports three different certificate sources, preflight checks and support bundle diagnostics will need to be tailored for each case. Letsencrypt certificates are of particular note as it is the most commonly used certificate source and has the greatest number of moving parts/areas of potential misconfiguration.

Potential checks/diagnostics

Preflight checks

• https://github.com/gitpod-io/gitpod/issues/11865
• Letsencrypt: Check that that the Issuer/ClusterIssue selector matches the Gitpod domain name (if the selector is specified)
• Letsencrypt: Check that cert-manager can manage DNS records for the configured zone
• Letsencrypt: Test the full certificate issuance process by creating a test certificate against letsencrypt-staging. (Heavyweight test, not something we'd run by default)
• User provided cert: verify that the private key matches the end certificate, and the end certificate issuer is the CA certificate
• External container registry: verify that containerd trusts the certificate presented by the external container registry

Support bundle checks

• The PEM-encoded https-certificates certificate. (Important to state - not the associated private key)
• Verify that kubelet/containerd can pull images from registry-facade
• Verify that a trivial prebuild workspace can be built
• Verify that certmanager can issue trivial certificates

Related

• https://github.com/gitpod-io/gitpod/issues/11005
• https://github.com/gitpod-io/gitpod/pull/11666
• https://github.com/gitpod-io/gitpod/issues/11811
• https://github.com/gitpod-io/gitpod/issues/10451 (note: supporting a pre-existing https-certificates object will require targeting preflight checks against that certificate)