AKS [Question] Override coreDNS default forward

Describe scenario I want to send all DNS requests to a specific servers like this:

  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . 10.0.0.0 10.0.0.1 {
          prefer_udp
        }
        cache 30
        loop
        reload
        loadbalance
    }

Instead of default configuration that looks like this:

  Corefile: |
    .:53 {
        errors
        ready
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import custom/*.override
    }
    import custom/*.server

I have attempted to use the coredns-custom ConfigMap, but I haven't had any success so far (it seems that /etc/resolv.conf continues to be used). I also read this closed issue (https://github.com/Azure/AKS/issues/2623), but there is no clear resolution mentioned.

Kubernetes version: v1.25.6 coreDNS version: mcr.microsoft.com/oss/kubernetes/coredns:v1.9.4

Question What is the correct way to override the default forward plugin?

Jul 13 '23 11:07 esierra-stratio

Current configuration:

coredns.yaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        ready
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import custom/*.override
    }
    import custom/*.server
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    ready\n    health\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n    import custom/*.override\n}\nimport custom/*.server\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kube-dns","kubernetes.io/cluster-service":"true"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2023-07-13T12:34:08Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
  name: coredns
  namespace: kube-system
  resourceVersion: "447"
  uid: 953c832c-0cbd-4036-a397-272c3efd7084

coredns-custom.yaml

apiVersion: v1
data:
custom.override: |
  forward . 10.0.0.0 10.0.0.1
  log
kind: ConfigMap
metadata:
creationTimestamp: "2023-07-13T12:34:04Z"
labels:
  addonmanager.kubernetes.io/mode: EnsureExists
  k8s-app: kube-dns
  kubernetes.io/cluster-service: "true"
name: coredns-custom
namespace: kube-system
resourceVersion: "18339"
uid: 08918cb9-242c-4ea3-acc8-aeac9dadba74

CoreDNS logs: kubectl logs -lk8s-app=kube-dns -n kube-system

root@esierra-custom-aks-control-plane:/# kw logs -lk8s-app=kube-dns -n kube-system
[INFO] 10.240.0.13:45167 - 57795 "A IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io.svc.cluster.local. udp 84 false 512" NXDOMAIN qr,aa,rd 177 0.000142802s
[INFO] 10.240.0.13:49651 - 4883 "AAAA IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io.cluster.local. udp 80 false 512" NXDOMAIN qr,aa,rd 173 0.000119202s
[INFO] 10.240.0.13:55555 - 6387 "A IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io.cluster.local. udp 80 false 512" NXDOMAIN qr,aa,rd 173 0.000107302s
[INFO] 10.240.0.13:55537 - 9406 "A IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io.sawv2s5tghhuldd4syau2ps5pb.ax.internal.cloudapp.net. udp 118 false 512" NXDOMAIN qr,aa,rd,ra 228 0.000101001s
[INFO] 10.240.0.13:43825 - 18835 "A IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io. udp 66 false 512" NOERROR qr,aa,rd,ra 130 0.000228204s
[INFO] 10.240.0.13:60349 - 30207 "AAAA IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io. udp 66 false 512" NOERROR qr,rd,ra 175 0.003520756s
[WARNING] No files matching import glob pattern: custom/*.server
[INFO] 10.240.0.9:44757 - 38503 "A IN google.es.default.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000212803s
[INFO] 10.240.0.9:52511 - 2628 "A IN google.es.sawv2s5tghhuldd4syau2ps5pb.ax.internal.cloudapp.net. udp 79 false 512" NXDOMAIN qr,rd,ra 189 0.004365361s
[INFO] 10.240.0.9:39266 - 20633 "A IN google.es. udp 27 false 512" NOERROR qr,rd,ra 52 0.006884197s
[INFO] 10.240.0.13:50139 - 25838 "AAAA IN esierra-custom-fz3zwghh.hcp.westeurope.azmk8s.io.sawv2s5tghhuldd4syau2ps5pb.ax.internal.cloudapp.net. udp 118 false 512" NXDOMAIN qr,rd,ra 228 0.0057808s
[WARNING] No files matching import glob pattern: custom/*.server
[INFO] 10.240.0.9:55997 - 41364 "A IN google.es.svc.cluster.local. udp 45 false 512" NXDOMAIN qr,aa,rd 138 0.000204904s
[INFO] 10.240.0.9:43117 - 63734 "A IN google.es.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000135202s
[INFO] 10.240.0.9:49539 - 16150 "A IN google.es.default.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000208404s
[INFO] 10.240.0.9:52728 - 40851 "A IN google.es.svc.cluster.local. udp 45 false 512" NXDOMAIN qr,aa,rd 138 0.000123802s
[INFO] 10.240.0.9:58774 - 23016 "A IN google.es.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000079401s
[INFO] 10.240.0.9:59504 - 7211 "A IN google.es.sawv2s5tghhuldd4syau2ps5pb.ax.internal.cloudapp.net. udp 79 false 512" NXDOMAIN qr,rd,ra 189 0.007138122s
[INFO] 10.240.0.9:52895 - 45354 "A IN google.es. udp 27 false 512" NOERROR qr,rd,ra 52 0.018599516s
[WARNING] No files matching import glob pattern: custom/*.server

Executing the following command: kubectl exec -it dnsutils -- nslookup google.es

Server:		192.168.0.10
Address:	192.168.0.10#53

Non-authoritative answer:
Name:	google.es
Address: 216.58.214.3

Jul 13 '23 13:07 esierra-stratio

Related to: https://github.com/Azure/AKS/issues/3661 https://github.com/Azure/AKS/issues/3232

Jul 13 '23 14:07 esierra-stratio

Any progress here?

Oct 11 '23 06:10 esierra-stratio

@esierra-stratio does this documentation help?

Jan 04 '24 23:01 RooMaiku

Action required from @Azure/aks-pm