AKS [Feature] kube-dns service not exposing metric port?

Describe scenario I have noticed that on my AKS clusters running on v1.24.9 the kube-dns service in the kube-system namespace is not exposing the coredns pod metrics. Thus the servicemonitor which is deployed by the prometheus-operator chart is not able to collect coredns metrics.

apiVersion: v1
kind: Service
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: kube-dns
  namespace: kube-system
spec:
  clusterIP: 10.0.0.10
  clusterIPs:
  - 10.0.0.10
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
  selector:
    k8s-app: kube-dns

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
    version: v20
  name: coredns
  namespace: kube-system
spec:
...
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: mcr.microsoft.com/oss/kubernetes/coredns:v1.9.3
        imagePullPolicy: IfNotPresent
        ...
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
...

From what I can see the official kubernetes manifest includes the metrics exposure on the kube-dns service. https://github.com/kubernetes/kubernetes/blob/v1.24.9/cluster/addons/dns/coredns/coredns.yaml.base

Describe the solution you'd like I would like to see AKS expose the metric directly on the kube-dns service or make it enable via a parameter via the AKS cli. Otherwise I do have to make this change on >10 clusters manually every time I redeploy them.

Apr 05 '23 13:04 rgarcia89

We are running into the same issue. I'm pretty sure this was exposed in the past (since we had alerts based on the metric).

May 14 '23 10:05 flo-02-mu

@flo-02-mu in case you are using the kube-prometheus-stack. I have just added a service to the aks platform jsonnet definition that will be created add used to scrape the coredns metrics.

https://github.com/prometheus-operator/kube-prometheus/pull/2107#event-9304184829

May 23 '23 08:05 rgarcia89

AKS runs multiple coredns pods behind the kube-dns service. If you scrape metrics from the service, the metrics might be inconsistent, because they might be from different pods. You can scrape metrics from pods to get consistent metrics with pod name in the dimension.

Jan 08 '24 23:01 robbiezhang

@robbiezhang that why we are using a headless service 😉

Jan 09 '24 06:01 rgarcia89

Action required from @aritraghosh, @julia-yin, @AllenWen-at-Azure