Connection probes made to be shown in status of GitHubIdentityProvider and WebhookAuthenticator do not respect HTTPS_PROXY env var

[cfryanr]

What happened?

If a cluster is firewalled in such a way that the Pinniped Concierge/Supervisor pods are prevented from making direct connections to the GitHub API (for GitHubIdentityProvider on the Supervisor) or to a webhook provider (for WebhookAuthenticator on the Concierge), and are instead expected to make connections using a web proxy defined by HTTPS_PROXY and NO_PROXY env vars on those pods, then GitHubIdentityProviders and WebhookAuthenticators will not become Ready or be usable.

This is because when we enhanced these custom resources to show much more detailed status to help operators better debug/understand their configurations, we used tls.Dial to implement the connection probes. However, tls.Dial does not respect the proxy settings.

Dialing was used because it has the advantage of testing connectivity and TLS negotiation without actually completing any http request, so it will avoid showing up in the webhook server's logs as a failed authentication. However, we didn't consider the proxy settings at that time.

What did you expect to happen?

These connection probes for status should respect the HTTPS_PROXY and NO_PROXY env vars set on the Concierge and/or Supervisor pods.

What is the simplest way to reproduce this behavior?

The latest version of Kind now supports NetworkPolicy, so it is now much easier to firewall certain traffic for specific pods on a Kind cluster, and therefore much easier to reproduce and test this behavior.

What else is there to know about this bug?

For WebhookAuthenticators, this behavior was probably introduced in v0.30.0, which is when we added detailed status to that custom resource. For GitHubIdentityProvider, it has always had this behavior since it was introduced in v0.31.0.