Let the proxy pod expose Prometheus metrics

[consideRatio]

trafficstars

With configurable-http-proxy version 4.4.0 just released, there is support for exposing prometheus metrics on a given port.

But, it is not obvious to me where to add annotations to help prometheus scrape this metrics.

• The CHP/proxy pod should run in a single replica.
• The CHP/proxy pod is exposed behind the k8s service named proxy-http if autohttps is enabled, or behind proxy-public if it isn't.
• The hub pod's k8s service is annotated so the hub pod is scraped.

Option 1: We could apply the prometheus scraping annotations to either the proxy-http service or the proxy-public service depending on where its located. Option 2: We could apply the prometheus scraping annotations to the proxy pod itself

Those are the viable options I think, and I'm not sure what makes sense or how the details work out.

Action points

• [ ] Configure --metrics-port=<some-port> when starting configurable-http-proxy
• [ ] Decide where to configure scraping annotations
  • [ ] Configure annotations

Related

https://github.com/jupyterhub/configurable-http-proxy/pull/314

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/714a3d2fb9c39aa696b67d2dee1840acb74b4bce/jupyterhub/templates/hub/service.yaml#L7-L19

[a3626a]

is it possible just to allow configuring extraPorts for the chp container? I can expose prometheus metrics by

• adding extraCommandLineFlags to proxy.chp
• adding extraPorts to proxy.service
• AND adding ports to chp container by kubectl. (not helm)

  proxy:
    chp:
      extraCommandLineFlags:
      - "--metrics-port=8080"
    service:
      extraPorts:
      - name: metric
        port: 8080
        targetPort: metric

apiVersion: apps/v1
kind: Deployment
metadata: ...
spec:
  ...
  template:
    metadata: ...
    spec:
      ...
      containers:
      - command:
        - configurable-http-proxy
        ...
        - --metrics-port=8080
        ...
        ports:
        - containerPort: 8000
          name: http
          protocol: TCP
        - containerPort: 8001
          name: api
          protocol: TCP
        // this should be able to added by helm value.
        - containerPort: 8080
          name: metric
          protocol: TCP
        ...

additionalScrapeConfigs for kube-prometheus-stack

      additionalScrapeConfigs:
        - job_name: jupyterhub
          metrics_path: "/hub/metrics"
          scheme: https
          static_configs:
            - targets:
                - ...
          authorization:
            credentials_file: "/etc/prometheus/secrets/jupyterhub/token"
        - job_name: proxy
          metrics_path: "/metrics"
          scheme: http
          static_configs:
            - targets:
                - jupyterhub-proxy-public.jupyter-hub.svc.cluster.local:8080