[VC-34401] Add metrics settings to the Helm chart

[wallrj]

In https://github.com/jetstack/jetstack-secure/pull/341 @tfadeyi added a metrics server to the agent. In this PR I've made the minimum viable changes to allow that metrics server to be queried by Prometheus, when the agent is installed by Helm in a Kubernetes cluster.

• I have chosen to only update the venafi-kubernetes-agent chart, because I believe the jetstack-secure agent is deprecated / retired.
• I decided not to make the metrics server port configurable. In csi-driver and approver-policy etc it is configurable, to allow users to change it in case it clashes with some other sidecar container that might be injected in the pod. If it becomes necessary, we can make the port configurable in a followup PR.
• I decided not to add any E2E tests...because there weren't any existing tests to use as examples.

🔗 FYI I recently made similar changes to cert-manager/csi-driver

• https://github.com/cert-manager/csi-driver/pull/271

Testing

• Create cluster

kind create cluster

• Install agent

helm upgrade venafi-kubernetes-agent ./deploy/charts/venafi-kubernetes-agent \
    --install \
    --create-namespace \
    --namespace venafi

• Fetch metrics directly

POD_NAME=$(kubectl get pod -n venafi -l app.kubernetes.io/instance=venafi-kubernetes-agent -o jsonpath='{ .items[0].metadata.name }')
kubectl get --raw "/api/v1/namespaces/venafi/pods/${POD_NAME}:8081/proxy/metrics" | grep HELP

...
# HELP go_info Information about the Go environment.
...
# HELP process_open_fds Number of open file descriptors.
...
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.

• Install kube-prometheus-stack

# values.kube-prometheus-stack.yaml
alertmanager:
  enabled: false

grafana:
  enabled: true

nodeExporter:
  enabled: false

# Enable discovery of all ServiceMonitor and PodMonitor resources
# https://github.com/prometheus-community/helm-charts/issues/1911#issuecomment-1106559031
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

helm upgrade -i default kube-prometheus-stack \
      --repo https://prometheus-community.github.io/helm-charts \
      --install \
      --namespace prometheus \
      --create-namespace \
      --values values.kube-prometheus-stack.yaml \
      --wait

• Enable the venafi-kubernetes-agent PodMonitor

helm upgrade venafi-kubernetes-agent ./deploy/charts/venafi-kubernetes-agent \
    --install \
    --create-namespace \
    --namespace venafi \
    --set metrics.podmonitor.enabled=true

• Connect to Grafana and import dashboards

kubectl port-forward -n prometheus deployments/default-grafana 3000

http://localhost:3000/d/ypFZFgvmz/go-processes (username admin, password prom-operator)

Example Dashboards

To import the dashboard, go to http://localhost:3000/dashboards and "New" → "Import", and paste the following dashboard URL and click "Load":

• https://grafana.com/grafana/dashboards/6671-go-processes/