[kube-prometheus-stack] Default ingress value pathType: ImplementationSpecific does't work on cilium ingress

[adippl]

Describe the bug a clear and concise description of what the bug is.

prometheus-kube-stack by default deploys prometheus ingress with pathType: ImplementationSpecific. This this doesn't work with cilium ingress because ImplementationSpecific pathType acts as exact match. Changing setting value pathType: Prefix solves the issue creates working ingress. Please consider changing the default values or add a warning talking about this issue in the comments of values file.

The same issue in cilium repo. https://github.com/cilium/cilium/issues/30119

What's your helm version?

version.BuildInfo{Version:"v3.11", GitCommit:"", GitTreeState:"", GoVersion:"go1.21.5"}

What's your kubectl version?

Client Version: v1.28.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.2

Which chart?

prometheus-kube-stack

What's the chart version?

56.6.2

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

fullnameOverride: "kps"
prometheus:
  service:
    type: LoadBalancer
  networkPolicy:
    enabled: false

    flavor: kubernetes
  ingress:
    enabled: true
    ingressClassName: cilium
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    labels: {}
    hosts:
      - prometheus.DOMAIN
    path: /.*
    tls:
    - hosts:
        - prometheus.DOMAIN
      secretName: prometheus.DOMAIN
    pathType: Prefix
  prometheusSpec:
    priorityClassName: "high-priority"
    externalLabels:
      cluster: k8s3
    retention: 30d
    replicas: 1
    podAntiAffinity: "hard"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: rook-ceph-block-ssd
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 25Gi
    resources:
     requests:
       cpu: 100m
       memory: 768Mi
     limits:
       cpu: 1000m
       memory: 2048Mi
    priorityClassName: "high-priority"

Enter the command that you execute and failing/misfunctioning.

curl https://prometheus.<<<MYDOMAIN>>>/graph -v
*   Trying 10.0.5.98:443...
* Connected to prometheus.<<<MYDOMAIN>>> (10.0.5.98) port 443
* ALPN: curl offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=prometheus.<<<MYDOMAIN>>>
*  start date: Jan  4 17:58:15 2024 GMT
*  expire date: Apr  3 17:58:14 2024 GMT
*  subjectAltName: host "prometheus.<<<MYDOMAIN>>>" matched cert's "prometheus.<<<MYDOMAIN>>>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /graph HTTP/1.1
> Host: prometheus.<<<MYDOMAIN>>>
> User-Agent: curl/8.4.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 404 Not Found
< date: Fri, 05 Jan 2024 14:24:01 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host prometheus.<<<MYDOMAIN>>> left intact

Anything else we need to know?

No response

[pierreozoux]

The ingress object looks like this:

    - host: alertmanager.mycluster.net
        paths:
          path: /
          pathType: ImplementationSpecific

I really think it could be replaced with Prefix without troubles. It would also help to enforce mitigation for this CVE-2022-4886, it is recommended to forbid ImplementationSpecific as a cluster admin.

relates to https://github.com/cert-manager/cert-manager/issues/6805#issuecomment-2522760035

For instance, this is the fix commit in OpenStack.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[ashtonian]

Still an issue.

[mydoomfr]

I am using Cilium 1.17.2 with Envoy Ingress and kube-prometheus-stack 70.3.0, and I am not experiencing this issue anymore.

spec:
  ingressClassName: cilium
  rules:
  - host: prometheus.homelab.local
    http:
      paths:
      - backend:
          service:
            name: monitoring-kube-prometheus-prometheus
            port:
              number: 9090
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - prometheus.homelab.local
    secretName: prometheus-tls

$ curl https://prometheus.homelab.local
<a href="/query">Found</a>.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.