[kube-prometheus-stack] Default ingress value pathType: ImplementationSpecific does't work on cilium ingress

[adippl]

Describe the bug a clear and concise description of what the bug is.

prometheus-kube-stack by default deploys prometheus ingress with pathType: ImplementationSpecific. This this doesn't work with cilium ingress because ImplementationSpecific pathType acts as exact match. Changing setting value pathType: Prefix solves the issue creates working ingress. Please consider changing the default values or add a warning talking about this issue in the comments of values file.

The same issue in cilium repo. https://github.com/cilium/cilium/issues/30119

What's your helm version?

version.BuildInfo{Version:"v3.11", GitCommit:"", GitTreeState:"", GoVersion:"go1.21.5"}

What's your kubectl version?

Client Version: v1.28.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.2

Which chart?

prometheus-kube-stack

What's the chart version?

56.6.2

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

fullnameOverride: "kps"
prometheus:
  service:
    type: LoadBalancer
  networkPolicy:
    enabled: false

    flavor: kubernetes
  ingress:
    enabled: true
    ingressClassName: cilium
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    labels: {}
    hosts:
      - prometheus.DOMAIN
    path: /.*
    tls:
    - hosts:
        - prometheus.DOMAIN
      secretName: prometheus.DOMAIN
    pathType: Prefix
  prometheusSpec:
    priorityClassName: "high-priority"
    externalLabels:
      cluster: k8s3
    retention: 30d
    replicas: 1
    podAntiAffinity: "hard"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: rook-ceph-block-ssd
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 25Gi
    resources:
     requests:
       cpu: 100m
       memory: 768Mi
     limits:
       cpu: 1000m
       memory: 2048Mi
    priorityClassName: "high-priority"

Enter the command that you execute and failing/misfunctioning.

curl https://prometheus.<<<MYDOMAIN>>>/graph -v
*   Trying 10.0.5.98:443...
* Connected to prometheus.<<<MYDOMAIN>>> (10.0.5.98) port 443
* ALPN: curl offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=prometheus.<<<MYDOMAIN>>>
*  start date: Jan  4 17:58:15 2024 GMT
*  expire date: Apr  3 17:58:14 2024 GMT
*  subjectAltName: host "prometheus.<<<MYDOMAIN>>>" matched cert's "prometheus.<<<MYDOMAIN>>>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /graph HTTP/1.1
> Host: prometheus.<<<MYDOMAIN>>>
> User-Agent: curl/8.4.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 404 Not Found
< date: Fri, 05 Jan 2024 14:24:01 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host prometheus.<<<MYDOMAIN>>> left intact

Anything else we need to know?

No response