External DNS tries to create records in a Peering zone

[mihail-i4v]

What happened: We have external-dns running in GKE. This has the --domain-filter flag set to sand1.example.us-west2.sandbox-gcp.qwerty.com.

In GCP we have two zones configured with DNS name set to sand1.example.us-west2.sandbox-gcp.qwerty.com. One zone is of private type. The other zone is of Peering type (https://cloud.google.com/dns/docs/zones/zones-overview#peering_zones).

We've added the --google-zone-visibility=private flag to external-dns in order to manage records only in the private DNS zone and ignore the peering DNS zone.

However, external-dns is still trying to manage both and is failing on creating records in the peering. Logs:

time="2022-08-10T08:33:10Z" level=info msg="All records are already up to date"
time="2022-08-10T08:34:11Z" level=info msg="Change zone: example-sandbox-us-east1-peer-example-sandbox-us-west2 batch #0"
time="2022-08-10T08:34:11Z" level=info msg="Add records: kubeflow-pipelines.sand1.example.us-west2.sandbox-gcp.qwerty.com. A [172.29.10.6] 300"
time="2022-08-10T08:34:11Z" level=error msg="googleapi: Error 400: Resource record sets for the zone 'example-sandbox-us-east1-peer-example-sandbox-us-west2' cannot be modified because it is a Peering zone., peeringChangeDisallowed"
time="2022-08-10T08:35:11Z" level=info msg="Change zone: example-sandbox-us-east1-peer-example-sandbox-us-west2 batch #0"
time="2022-08-10T08:35:11Z" level=info msg="Add records: kubeflow-pipelines.sand1.example.us-west2.sandbox-gcp.qwerty.com. A [172.29.10.6] 300"
time="2022-08-10T08:35:11Z" level=info msg="Add records: kubeflow-pipelines.sand1.example.us-west2.sandbox-gcp.qwerty.com. TXT [\"heritage=external-dns,external-dns/owner=example-sand1-003-external-dns,external-dns/resource=ingress/kubeflow-pipelines/kubeflow-pipelines-ingress-internal\"] 300"
time="2022-08-10T08:35:11Z" level=error msg="googleapi: Error 400: Resource record sets for the zone 'example-sandbox-us-east1-peer-example-sandbox-us-west2' cannot be modified because it is a Peering zone., peeringChangeDisallowed"
time="2022-08-10T08:36:12Z" level=info msg="Change zone: example-us-west2-sandbox-gcp-qwerty-com batch #0"
time="2022-08-10T08:36:12Z" level=info msg="Add records: kubeflow-pipelines.sand1.example.us-west2.sandbox-gcp.qwerty.com. A [172.29.10.6] 300"
time="2022-08-10T08:36:12Z" level=info msg="Add records: kubeflow-pipelines.sand1.example.us-west2.sandbox-gcp.qwerty.com. TXT [\"heritage=external-dns,external-dns/owner=example-sand1-003-external-dns,external-dns/resource=ingress/kubeflow-pipelines/kubeflow-pipelines-ingress-internal\"] 300"
time="2022-08-10T08:37:13Z" level=info msg="All records are already up to date"

What you expected to happen: External-dns manages only the private zone.

How to reproduce it (as minimally and precisely as possible):

• Create 2 zones in GCP - one private and one peering - with the same DNS Name.
• Set --domain-filter to be the DNS Name.
• Set --google-zone-visibility=private

Anything else we need to know?: External-dns fetches the zone in random order and eventually it succeeds to create a record in the correct zone.

Environment:

• External-DNS version (use external-dns --version): 0.11.0
• DNS provider: Cloud DNS (GCP)
• Others: