reverseproxy: Add upstreams healthy metrics

[david-szabo97]

Adds a new metric that tracks the health status of reverse proxy upstreams.

# HELP caddy_reverse_proxy_upstreams_healthy Health status of reverse proxy upstreams.
# TYPE caddy_reverse_proxy_upstreams_healthy gauge
caddy_reverse_proxy_upstreams_healthy{upstream="localhost:8080"} 1
caddy_reverse_proxy_upstreams_healthy{upstream="localhost:8081"} 0

Metric:

• Name: caddy_reverse_proxy_upstreams_healthy
• Type: gauge
• Values:
  • 1 - the upstream is healthy
  • 0 - the upstream is unhealthy
• Labels:
  • upstream - Address of the upstream

Example Grafana panel:

Partially resolves https://github.com/caddyserver/caddy/issues/4140

[CLAassistant]

All committers have signed the CLA.

[mholt]

Looks cool! I'm not familiar with Prometheus so I'll let our in-house expert take a look if he has a chance :smiley:

I'll review the usage of the reverse proxy internals though.

[mholt]

@hairyhenderson I know you're busy so I won't require a review from you to merge this! Let me know if you want me to hold off so you can review, otherwise I will probably just merge it in.

[mholt]

Great, thank you for the contribution @david-szabo97 !

[HeCorr]

I'm not a big fan of the hard-coded 10s delay between metric updates. I would personally like a shorter delay so metrics (and consequently alarms) are more precise, thus a way to configure that would be greatly appreciated! Awesome work nonetheless :)