External DNS adding then deleting the same record to etcd causing unnecessary writes

[eroji]

trafficstars

What happened: I noticed that with External DNS using etcd as backend for storing the DNS records, it is adding then deleting the same record over and over. If you have namespaces with a lot of services annotated, this will amplify the writes and ultimately fill up etcd, which then cases External DNS to crashloop.

What you expected to happen: External DNS should recognize that the record is current and not need to delete it.

How to reproduce it (as minimally and precisely as possible): The most reliable way to do this is if you have a service created, annotate it, apply it. Then update the service YAML and change the annotation. Then do a kubectl replace -f service.yaml -n <namespace>. This seems to cause the problem pretty consistently. I'm not super clear on the triggering condition.

time="2025-04-29T00:32:05Z" level=info msg="Delete key /skydns/com/example/svc/example-internal-datacenter/a-jaeger-agent"
time="2025-04-29T00:33:06Z" level=info msg="Add/set key /skydns/com/example/svc/example-internal-datacenter/a-jaeger-agent/679dd636 to Host=, Text=\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/observability/jaeger-agent\", TTL=0"
time="2025-04-29T00:33:06Z" level=info msg="Add/set key /skydns/com/example/svc/example-internal-datacenter/a-jaeger-agent-otlp/6cfe9e98 to Host=, Text=\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/observability/jaeger-agent-otlp\", TTL=0"
time="2025-04-29T00:33:06Z" level=info msg="Generating new prefix: (73961a4f)"
time="2025-04-29T00:33:06Z" level=info msg="Delete key /skydns/com/example/svc/example-internal-datacenter/jaeger-agent/default"
time="2025-04-29T00:33:06Z" level=info msg="Add/set key /skydns/com/example/svc/example-internal-datacenter/jaeger-agent/73961a4f to Host=10.2.111.110, Text=\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/observability/jaeger-agent\", TTL=0"
time="2025-04-29T00:33:06Z" level=info msg="Generating new prefix: (24c302c7)"
time="2025-04-29T00:33:06Z" level=info msg="Delete key /skydns/com/example/svc/example-internal-datacenter/jaeger-agent-otlp/default"
time="2025-04-29T00:33:06Z" level=info msg="Add/set key /skydns/com/example/svc/example-internal-datacenter/jaeger-agent-otlp/24c302c7 to Host=10.2.111.111, Text=\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/observability/jaeger-agent-otlp\", 

Anything else we need to know?: To stop External DNS to repeatedly delete then add the records, I went and deleted the services with kubectl delete -f service.yaml. Then added them back again. After which it stopped this loop.

time="2025-04-29T00:38:07Z" level=info msg="All records are already up to date"
time="2025-04-29T00:39:08Z" level=info msg="All records are already up to date"
time="2025-04-29T00:40:08Z" level=info msg="All records are already up to date"
time="2025-04-29T00:41:09Z" level=info msg="All records are already up to date"
time="2025-04-29T00:42:09Z" level=info msg="All records are already up to date"
time="2025-04-29T00:43:10Z" level=info msg="All records are already up to date"

Environment:

• External-DNS version (use external-dns --version): v0.16.1
• DNS provider: local etcd (coreDNS)
• Others: