[External-DNS] Sync mode does not delete/update DNS entries when changes are made on kubernetes side

[baptistebalmon]

What happened: The externalDNS has been configured this way :

image: registry.k8s.io/external-dns/external-dns:v0.14.0
args:
- --policy=sync
- --source=ingress
- --source=service
- --annotation-filter=managed.by=external-dns
- --txt-owner-id=<rke cluster name>-ext-dns
- --txt-prefix=dbaas.

I'm using RFC2136 mode with PowerDNS.

this way, only services and ingresses that have the annotation "managed.by=external-dns" will be managed. also txt-owner-id allows to identify the RKE Cluster on which the DNS entry should lead to the txt-prefix allows to create CNAME entries (for ingresses mainly)

The creation works well :

• 2 TXT records
• A/CNAME record

Here is an example of a kubernetes service that needs to be created :

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: toto.example.org
    external-dns.alpha.kubernetes.io/ttl: "3600"
    loadbalancer.openstack.org/load-balancer-address: X.X.X.X
    managed.by: external-dns
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  sessionAffinity: None
  type: LoadBalancer

When I delete the kubernetes service or ingress, externalDNS does not delete them on the DNS. When I update the kubernetes service or ingress, externalDNS does not update them on the DNS but creates a new entry.

What you expected to happen: On the kubernetes resources (svc/ingress) deletion, cleaning the corresponding A/CNAME and TXT records from the DNS On the kubernetes resources (svc/ingress) update, updating the corresponding A/CNAME and TXT records from the DNS.

How to reproduce it (as minimally and precisely as possible):

1. Deploy externalDNS version 0.14.0
2. Create kubernetes services
3. Check that the records are created on the DNS
4. Delete the kubernetes service
5. Check that the records are not deleted on the DNS

Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): 0.14.0
• DNS provider: PowerDNS
• Others: RFC2136

[leonardocaylent]

@baptistebalmon Did you check if this is working correctly with v0.13.6? Can you attach the external-dns pod logs in Debug level?

[baptistebalmon]

Hi @leonardocaylent I've tried to downgrade the externalDNS image version to v0.13.6. The issue is still the same. Kubernetes resources are deleted but DNS records are not. Please find the log in debug mode.

time="2024-05-07T13:44:03Z" level=debug msg="axfr is disabled" time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from ingress: toto/haproxy-stats: [toto.org 3600 IN CNAME vip.org []]" time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from service: titi/titi-pgbouncer: [titi.org 3600 IN A X.X.X.X []]" time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto-replicas.org 3600 IN A X.X.X.X []]" time="2024-05-07T13:44:03Z" level=debug msg="ApplyChanges (Create: 9, UpdateOld: 0, UpdateNew: 0, Delete: 0)" time="2024-05-07T13:44:03Z" level=debug msg="Processing batch 0 of create changes" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=toto.org 3600 IN CNAME vip.org []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: toto.org 3600 CNAME vip.org" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=toto-replicas.org 3600 IN A X.X.X.X []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: toto-replicas.org 3600 A X.X.X.X" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=titi.org 3600 IN A X.X.X.X []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: titi.org 3600 A X.X.X.X" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.toto.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.toto.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats"" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.cname-toto.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.cname-toto.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats"" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy"" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.a-toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy"" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer"" time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.a-titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []" time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.a-titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer"" time="2024-05-07T13:44:03Z" level=debug msg=SendMessage time="2024-05-07T13:44:03Z" level=debug msg=SendMessage.success

time="2024-05-07T13:49:03Z" level=debug msg="axfr is disabled" time="2024-05-07T13:49:03Z" level=debug msg="Endpoints generated from service: titi/titi-pgbouncer: [titi.org 3600 IN A X.X.X.X []]" time="2024-05-07T13:49:03Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto-replicas.org 3600 IN A X.X.X.X []]" time="2024-05-07T13:49:03Z" level=debug msg="ApplyChanges (Create: 6, UpdateOld: 0, UpdateNew: 0, Delete: 0)" time="2024-05-07T13:49:03Z" level=debug msg="Processing batch 0 of create changes" time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=toto-replicas.org 3600 IN A X.X.X.X []" time="2024-05-07T13:49:03Z" level=info msg="Adding RR: toto-replicas.org 3600 A X.X.X.X" time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=titi.org 3600 IN A X.X.X.X []" time="2024-05-07T13:49:03Z" level=info msg="Adding RR: titi.org 3600 A X.X.X.X" time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []" time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy"" time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []" time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.a-toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy"" time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []" time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer"" time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.a-titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []" time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.a-titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer"" time="2024-05-07T13:49:03Z" level=debug msg=SendMessage time="2024-05-07T13:49:03Z" level=debug msg=SendMessage.success

Thank you for your help.

[leonardocaylent]

@baptistebalmon I can see that your plan is not detecting anything to delete. I need to ask you 3 more questions: 1)Has this ever worked for you before? 2)Can you test again with the versions v0.12.0 and v0.10.0? 3)If you manually delete the records external-dns creates them again on the next round?

If you can add the creation logs that could also help.

Thank you for reporting this issue

[baptistebalmon]

hi @leonardocaylent , thank you for helping me out on this. It seems that externalDNS only creates (as we can see in the logs), I could never see UpdateOld, UpdateNew or Delete. It explains why it recreates every entry but does not delete any. Is it a normal behavior ?

1. I guess that the delete worked once first time that I installed externalDNS. The only difference is that I didn't configure the annotation filter on the records to be managed by externalDNS. I've tried to remove this configuration but it still doesn't work.

2. I've tried with both v0.12.0 and v0.10.0 and I've got same result.

3. If I manually delete the records, they are created again on the next round.

Here are the logs of creation : time="2024-05-29T08:38:01Z" level=debug msg="axfr is disabled" time="2024-05-29T08:38:01Z" level=debug msg="Endpoints generated from ingress: toto/haproxy-stats: [toto-haproxy-stats.io 3600 IN CNAME priv-toto.io []]" time="2024-05-29T08:38:01Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto.io 3600 IN A X.X.X.X []]" time="2024-05-29T08:38:01Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy-replicas: [toto-replicas.io 3600 IN A X.X.X.X []]" time="2024-05-29T08:38:01Z" level=debug msg="ApplyChanges (Create: 9, UpdateOld: 0, UpdateNew: 0, Delete: 0)" time="2024-05-29T08:38:01Z" level=debug msg="Processing batch 0 of create changes" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=toto-haproxy-stats.io 3600 IN CNAME priv-mtdbaas01stg.io []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: toto-haproxy-stats.io 3600 CNAME priv-mtdbaas01stg.io" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=toto-replicas.io 3600 IN A X.X.X.X []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: toto-replicas.io 3600 A X.X.X.X" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=toto.io 3600 IN A X.X.X.X []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: toto.io 3600 A X.X.X.X" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.toto-haproxy-stats.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.toto-haproxy-stats.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats"" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.cname-toto-haproxy-stats.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.cname-toto-haproxy-stats.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats"" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas" []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.toto-replicas.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas"" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas" []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.a-toto-replicas.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas"" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.toto.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy" []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.toto.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy"" time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.a-toto.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy" []" time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.a-toto.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy"" time="2024-05-29T08:38:01Z" level=debug msg=SendMessage time="2024-05-29T08:38:01Z" level=debug msg=SendMessage.success

[baptistebalmon]

Hi again @leonardocaylent , problem solved

I've found that AXFR is mandatory to update or delete DNS records.

The TSIG key that I was using was only configured with DYNUPDATE. I've added :

• AXFR on the TSIG key
• rfc2136-tsig-axfr flag to true on the externalDNS container

Updates and Deletes work now fine.

Thank you for the help.