feat: add new flags to allow migration of OwnerID
Description
This pull requests aims to complete the work initiated in these PR : https://github.com/kubernetes-sigs/external-dns/pull/2466 and https://github.com/kubernetes-sigs/external-dns/pull/3631
Most credits go to the creators of the initial work, as I mostly just rebased their work on the current state of master
Initial PR description:
When changing --txt-owner-id on an existing external-dns resource, it does not update the existing TXT records it owns, therefore losing ownership. Meaning that we have to manually delete the records in order to have external-dns take ownership again. To solve this problem, I added the ability to update the original txt-owner by setting -- migrate-txt-owner to overwrite the old txt-owner. I have successfully modified thousands of pieces of data using this code, so far without any bugs
As for me I've successfully managed records without any issue using the newly added flags on the OVH provider If anyone is willing to try these changes on other providers I'd be happy to troubleshoot their issues
Regarding the documentation I'm not sure either in which category these changes are relevant, so taking any recommendations on that
Note: this is my first contribution to such a project and first time using Go aswell, I've went through the contribution guidelines but if I missed something please let me know so that I can fix it
Fixes #2036
Checklist
- [x] Unit tests updated
- [x] End user documentation updated
Welcome @troll-os!
It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.
You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.
You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.
You may want to refer to our testing guide if you run into trouble with your tests not passing.
If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!
Thank you, and welcome to Kubernetes. :smiley:
Hi @troll-os. Thanks for your PR.
I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Thanks @troll-os. It looks good. Would you please add some documentation on it ?
@mloiseleur Sure !
Where do you think it would fit the best? TXT registry ?
With this PR, it works only with TXT registry, so yes.
/ok-to-test
/lgtm
/assign @Raffo
@troll-os answered to your comments. Could you please also provide a way to test this manually with manifests and kubectl commands? I would be eager to verify myself that the migration works and then keep an end to end test around.
So I had to make a change for the OVH provider to ensure record targets are properly formatted, I ran another batch of test to migrate from the default owner and for some reason, records returned by the OVH API don't have their targets formatted like ExternalDNS expects
Meaning that we couldn't map the record ID and the deletion would fail
I don't know why this happens on a specific zones, the other zones covered seem fine... When I make direct calls to the OVH API the formatting is good...
I didn't forget your query for manifests to tests the changes @Raffo, I'll make them ASAP
So I did a little of homework and made this work with Gandi with a domain I own for personal use and minikube
Steps used to reproduce on another setup than my work dev and prod environments :
make buildon the branch of this PR (eventually push to a repo you own)- Setup Minikube with Docker
minikube enable addons ingress- Use following manifests (update variables to ensure your tests cases):
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
external-dns.alpha.kubernetes.io/hostname: helloworld.whatever.tld
spec:
selector:
app: nginx
type: LoadBalancer
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
spec:
ingressClassName: nginx
rules:
- host: helloworld.whatever.tld
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
replicas: 1
selector:
matchLabels:
app: external-dns
strategy:
type: Recreate
template:
metadata:
labels:
app: external-dns
spec:
containers:
- name: external-dns
image: yourRegistry.com/external-dns:v-someBuildNumber # <<<<<<<<<<<<
args:
- "--txt-prefix=%{record_type}-"
- "--txt-cache-interval=15m"
- "--log-level=info"
- "--log-format=text"
- "--txt-owner-id=old-owner"
- "--policy=sync"
- "--provider=youDecide" # <<<<<<<<<<<<<<<<
- "--registry=txt"
- "--interval=1m"
- "--domain-filter=whatever.tld"
- "--source=ingress"
env:
- name: whatever_env_var_provider_needs
value: "hello"
- Check your records, the A and TXT should be there, owned by "old-owner"
- Apply following manifest to make migration :
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
replicas: 1
selector:
matchLabels:
app: external-dns
strategy:
type: Recreate
template:
metadata:
labels:
app: external-dns
spec:
containers:
- name: external-dns
image: yourRegistry.com/external-dns:v-someBuildNumber # <<<<<<<<<<<<
args:
- "--txt-prefix=%{record_type}-"
- "--txt-cache-interval=15m"
- "--log-level=info"
- "--log-format=text"
- "--txt-owner-id=new-owner"
- "--from-txt-owner=old-owner"
- "--migrate-txt-owner"
- "--policy=sync"
- "--provider=youDecide" # <<<<<<<<<<<<
- "--registry=txt"
- "--interval=1m"
- "--domain-filter=whatever.tld"
- "--source=ingress"
env:
- name: whatever_env_var_provider_needs
value: "hello"
- Check TXT record again, everything should be up to date
Also I tried to follow the CoreDNS setup from the external-dns docs to have a non "external" provider dependent way to tests but didn't manage to make it work for some reason and I didn't want to spend too much time on it (probably a minikube issue) knowing that it worked with a classic provider
Hello
Hope you guys had a great end to 2024 and best wishes for 2025 :)
Do you know if you'll be able to test this we the information I provided ?
We've used this code in prod on our side without issues for OVH and confirmed it works for Gandi, in theory should be the same for others aswell. I might be able to test on Azure DNS soon
@Dadeos-Menlo: changing LGTM is restricted to collaborators
In response to this:
I appreciate that the proposed changes are predominantly the work of others; however it shouldn't be necessary to modify anything above the
registry.TXTRegistrylevel (i.e. neithercontroller.Controllernorplan.Plan).The registry system acts as a wrapper around various provider implementations. The TXTRegistry works by generating DNS TXT records describing the set of Labels associated with a given
Endpointand injecting any mutations required to create/update/delete those DNS TXT records as part of the ApplyChanges(…) processing. Any such DNS TXT records are subsequently filtered during the retrieval of DNS records from the provider, via the Records(…) function, and the information that they convey (i.e. theLabels) merged into the correspondingEndpointobject describing the real (i.e. non-TXT) DNS record being managed. Therefore migration of aLabelvalue (e.g. Owner) from an old value to a new value should be achievable entirely within theTXTRegistryimplementation.The required processing would involve identifying DNS TXT records describing ownership associated with any provided old owner value and treating them as if they had been associated with the new/current owner during the filtering stage and then ensuring that any DNS TXT records associated with the old owner value are replaced by DNS TXT records associated with the new/current owner value during the generation stage.
There is perhaps also a case for providing wider support for migrating the configuration of the
TXTRegistry; for example, also supporting the migration of txt-prefix and txt-suffix configurations.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@troll-os Do you think you can address review comments from @Dadeos-Menlo ?
Hi @mloiseleur
Yes I noted his feedback, I didn't find time to work on this yet, but definitely not set aside
Are there any deadlines on your side that I should take into account ?
@troll-os nope, just wanting to check if you were still here and did see the comments
Any timeline on when this PR will be merged ?
The committers listed above are authorized under a signed CLA.
- :white_check_mark: login: troll-os / name: troll-os (cea2b7735df6b4f2f9f1ce7b0d9ce0c3b417fe17, 79b72793ccb4baa96d1363f41f0e26c5cd3eaf08, 4a91fecf36bc0d16dc865f8c714796ec9418d68a)
- :white_check_mark: login: troll-os / name: Thibaut R (6660700f3ab9d30a695cf1329d67d136f2714855, 4dec559c85936c3dc83cfa0d55524130815c3ea7, 41c9a79791ea4400599051f06e394f530a5ae26f, ed5d0fd1d010ca1362f803b480e9b80922eb818d, f7a450348731158813633c4c01a2e5cf27ca6797, 7edd2d879dc310922f3573e6d25eff2265232415, 16ea3eeb0aa2343d7bd062ca14281b9eb1d4a7c9, 0c0928b51641e0b7252279b497d9be3b59e3bc57, b91622122a6250bef92cbf95e4f0c7f4b90ff03c, fe7435cda06994d5ccdd3c7f664d9926a3eb01b3, 1430bfaf0eae8ca41804bb76d55c0a79d46f5e20, 25a733e0e08909e6bb819d3bebb1dcd615b66040, 8eaf4153efb0ae2328e463dd6ec7869ab2b36100, 104ab1f686f9294c793cd1f6901cd5774974f95c, acdd1a6d49b2ca9bdc774e22a5acd8013536bcb1)
Damn I think I've made a mistake with my rebase...
I'll wipe every commits and make a fresh push with the latest feedbacks only
Again sorry for the delay in this PR handling, I'll have some time off and I'll deal with it the next few days
PR has been auto closed by push force to reset a clean branch state
I'm currently repicking the work I've done on TXTRegistry with the feedback from Dadeos and I'll have to rerun all the tests of course because my branch diverged a lot from when I've opened this PR
Hi @Dadeos-Menlo
I'm failing to see how to make it happen without updating at least the Plan (but in reality the Controller too)
I tried to wrap my head around the issue but in practice when instantiated in Kube, the bit of code updated in the txt Registry implementation, which does update the records accordingly, as the additional tests suggest, is not sufficient.
With the current state of the PR, the part at https://github.com/FiligranHQ/external-dns/blob/c481f3320cc53df8a702ada717299b7d22946542/plan/plan.go#L232 will result in the old records being skipped in the update process.
If you have any reco to make this work I'll be glad to take it, otherwise I might have to pursue with passing down the migration and old owner id value to the plan from the controller and add the relevant condition on migration enabled in the plan.
@troll-os It seems there is no answer from @Dadeos-Menlo. Since the implementation of this PR is quite straightforward, and it has been tested on multiple providers, so I think we can merge it. Would you please fix the linter issue about the trailing newline ?
@mloiseleur by any chance this one can be merged soon? we really need this kind of solution in our case. @troll-os
@nadavshanioptimove Once the PR is rebased and CI is green, we can do final review and merge it.
If @troll-os can do it, that would be great, otherwise feel free to open a new PR, based on this one.
Sorry for missing out
I'll try to complete this PR by the end of the week
By any chance could you as well share results of a smoke tests, need to make sure the feature is actually working
@ivankatliarchuk
By any chance could you as well share results of a smoke tests, need to make sure the feature is actually working
This is the result I get on Gandi after using the YAML file I shared earlier in this thread as an example + my latest push to run tests :
-
Initial deployment, creation of records with old owner flag
-
Log output of the run when the migration is applied
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm1.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm2.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm3.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint webmail.!!!REDACTED!!!.cloud 10800 IN CNAME webmail.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint www.!!!REDACTED!!!.cloud 10800 IN CNAME webredir.vip.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!!.cloud 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint helloworld.!!!REDACTED!!!.cloud 600 IN A 192.168.49.2 [] because owner id does not match, found: \"new-owner\", required: \"old-owner\""
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint helloworld.!!!REDACTED!!!.cloud 0 IN A 192.168.49.2 [] because owner id does not match, found: \"new-owner\", required: \"old-owner\""
time="2025-06-21T22:21:21Z" level=debug msg="Excluding domain !!!REDACTED!!! by domain-filter"
time="2025-06-21T22:21:21Z" level=debug msg="Excluding domain !!!REDACTED!!! by domain-filter"
time="2025-06-21T22:21:21Z" level=debug msg="Excluding domain !!!REDACTED!!! by domain-filter"
time="2025-06-21T22:21:21Z" level=info msg="Changing record" action=UPDATE record=helloworld ttl=600 type=A value=192.168.49.2 zone=!!!REDACTED!!!.cloud
time="2025-06-21T22:21:21Z" level=info msg="Changing record" action=UPDATE record=a-helloworld ttl=600 type=TXT value="\"heritage=external-dns,external-dns/owner=new-owner,external-dns/resource=ingress/default/nginx\"" zone=!!!REDACTED!!!.cloud
I0621 22:22:20.004014 1 transport.go:356] "Warning: unable to cancel request" roundTripperType="*instrumented_http.Transport"
time="2025-06-21T22:22:20Z" level=debug msg="Using cached records."
time="2025-06-21T22:22:20Z" level=debug msg="Endpoints generated from ingress: default/nginx: [helloworld.!!!REDACTED!!!.cloud 0 IN A 192.168.49.2 []]"
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm1.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm2.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm3.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!!.cloud 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint webmail.!!!REDACTED!!!.cloud 10800 IN CNAME webmail.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint www.!!!REDACTED!!!.cloud 10800 IN CNAME webredir.vip.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\""
time="2025-06-21T22:22:20Z" level=info msg="All records are already up to date"
- State of records in the provider after the run
@ivankatliarchuk
By any chance could you as well share results of a smoke tests, need to make sure the feature is actually working
This is the result I get on Gandi after using the YAML file I shared earlier in this thread as an example + my latest push to run tests :
- Initial deployment, creation of records with old owner flag
- Log output of the run when the migration is applied
time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm1.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm2.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm3.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint webmail.!!!REDACTED!!!.cloud 10800 IN CNAME webmail.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint www.!!!REDACTED!!!.cloud 10800 IN CNAME webredir.vip.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!!.cloud 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint helloworld.!!!REDACTED!!!.cloud 600 IN A 192.168.49.2 [] because owner id does not match, found: \"new-owner\", required: \"old-owner\"" time="2025-06-21T22:21:20Z" level=debug msg="Skipping endpoint helloworld.!!!REDACTED!!!.cloud 0 IN A 192.168.49.2 [] because owner id does not match, found: \"new-owner\", required: \"old-owner\"" time="2025-06-21T22:21:21Z" level=debug msg="Excluding domain !!!REDACTED!!! by domain-filter" time="2025-06-21T22:21:21Z" level=debug msg="Excluding domain !!!REDACTED!!! by domain-filter" time="2025-06-21T22:21:21Z" level=debug msg="Excluding domain !!!REDACTED!!! by domain-filter" time="2025-06-21T22:21:21Z" level=info msg="Changing record" action=UPDATE record=helloworld ttl=600 type=A value=192.168.49.2 zone=!!!REDACTED!!!.cloud time="2025-06-21T22:21:21Z" level=info msg="Changing record" action=UPDATE record=a-helloworld ttl=600 type=TXT value="\"heritage=external-dns,external-dns/owner=new-owner,external-dns/resource=ingress/default/nginx\"" zone=!!!REDACTED!!!.cloud I0621 22:22:20.004014 1 transport.go:356] "Warning: unable to cancel request" roundTripperType="*instrumented_http.Transport" time="2025-06-21T22:22:20Z" level=debug msg="Using cached records." time="2025-06-21T22:22:20Z" level=debug msg="Endpoints generated from ingress: default/nginx: [helloworld.!!!REDACTED!!!.cloud 0 IN A 192.168.49.2 []]" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm1.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm2.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN CNAME gm3.gandimail.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!!.cloud 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint !!!REDACTED!!! 10800 IN A !!!REDACTED!!! [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint webmail.!!!REDACTED!!!.cloud 10800 IN CNAME webmail.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=debug msg="Skipping endpoint www.!!!REDACTED!!!.cloud 10800 IN CNAME webredir.vip.gandi.net [] because owner id does not match, found: \"\", required: \"new-owner\"" time="2025-06-21T22:22:20Z" level=info msg="All records are already up to date"
- State of records in the provider after the run
because owner id does not match, found: "", required: "new-owner""
will that be dismissed later on when new owner is new-owner? also is the migration flag supportes older format of txt records without the type prefix?
This log was not introduced by this PR and only shown in debug
This is because I have manually created some records on this domain previously
This is basically a mechanism that makes external-dns ignore records it doesn't manage to avoid destruction or bad manipulation of the said records
06-21T22:22:20Z" level=debug msg="Skipping endpoint webmail.!!!REDACTED!!!.c
is the migration flag applied for the same instance? (meaning changing current instance external-dns's owner id) or is it migration from old instance to new instance with new owner id? because if it can be used within the migration of same instance, so in some point it should be in sync with the updated txt record. am i wrong?

