Prometheus stats should be exposed when deploying extauth as sidecar

[willowmck]

Is your feature request related to a problem? Please describe. When deploying extauth as a sidecar to the gateway, Prometheus stats exposed by extauth are not being scraped by glooe-prometheus-server. This is because the only annotations that are exposed are the current ones in the gateway deployment.

  Annotations:      prometheus.io/path: /metrics
                             prometheus.io/port: 8081
                             prometheus.io/scrape: true

Likewise, the configmap for prometheus would need to be modified to add a second job to pickup these new annotations. The current configmap has the following job defined.

    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: kubernetes_pod_name
      - action: drop
        regex: Pending|Succeeded|Failed
        source_labels:
        - __meta_kubernetes_pod_phase

Describe the solution you'd like

Since extauth exposes prometheus metrics on port 9091 we would need to modify the gateway deployment and add a second set of annotations.

Doing this through the current Helm chart of adding extauth as a sidecar should modify the gateway annotations and also modify the prometheus config map to scrape the new annotations.

Describe alternatives you've considered Another approach would be to document for the user how to modify the ConfigMap and Gateway deployment but this could lead to errors that we would need to handle via support.

Additional context The need to display latency metrics while analyzing extauth opa performance caused us to discover that Prometheus was not capturing extauth data when being deployed as a sidecar. This was noticed on Gloo Edge Enterprise 1.7.7 but could affect other versions.

[github-actions[bot]]

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.

[github-actions[bot]]

This issue has been closed due to no activity in the last 12 months.