external-dns icon indicating copy to clipboard operation
external-dns copied to clipboard
verified publisher icon kubernetes-sigs

OVHcloud API error (status code 404): Client::NotFound: \"Got an invalid (or empty) URL

Open feluxe opened this issue 6 months ago • 10 comments

What happened:

I upgraded from 1.16.1 and now get this error:

time="2025-10-20T15:04:33Z" level=error msg="Failed to do run once: soft error\nOVHcloud API error (status code 404): Client::NotFound: \"Got an invalid (or empty) URL\" (X-OVH-Query-Id: EU.ext-2.68f64f81.842220.afb3a70c1c04ffdb199cc36e15ad7a9b) (consecutive soft errors: 2)"

I tried with v1.17.0, v1.18.0 and v1.19.0 and get the same error on each.

I removed all DNS records manually and deployed a new k8s cluster, without luck.

No DNS records were created by external-dns at all.

What you expected to happen:

I didn't expect to get this error, and I expected external-dns to create DNS records for my ingresses.

How to reproduce it (as minimally and precisely as possible):

  • Create a DNS zone on OVH
  • Create a new cluster v1.33
  • Deploy ingress-nginx v1.13.3 (default config)
  • Wait for loadBalancer to get ready
  • Deploy external-dns v1.17.0 or newer with this config:

From the docs at https://github.com/kubernetes-sigs/external-dns/blob/v0.19.0/docs/tutorials/ovh.md

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.k8s.io/external-dns/external-dns:v0.18.0
        args:
        - --provider=ovh
        - --source=ingress
        - --policy=upsert-only
        - --registry=txt
        - --txt-owner-id=my-cluster-1--
        - --log-level=trace
        - --interval=5m
        - --txt-prefix=my-cluster-1--

        env:
        - name: OVH_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: provider-ovh
              key: OVH_ENDPOINT
        - name: OVH_APPLICATION_KEY
          valueFrom:
            secretKeyRef:
              name: provider-ovh
              key: OVH_APPLICATION_KEY
        - name: OVH_APPLICATION_SECRET
          valueFrom:
            secretKeyRef:
              name: provider-ovh
              key: OVH_APPLICATION_SECRET
        - name: OVH_CONSUMER_KEY
          valueFrom:
            secretKeyRef:
              name: provider-ovh
              key: OVH_CONSUMER_KEY

Pod log until the error (which is repeating)

time="2025-10-20T14:59:32Z" level=info msg="GitCommitShort=unknown, GoVersion=go1.24.4, Platform=linux/amd64, UserAgent=ExternalDNS/v20250626-v0.18.0"
time="2025-10-20T14:59:32Z" level=info msg="Instantiating new Kubernetes client"
time="2025-10-20T14:59:32Z" level=debug msg="apiServerURL: "
time="2025-10-20T14:59:32Z" level=debug msg="kubeConfig: "
time="2025-10-20T14:59:32Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2025-10-20T14:59:32Z" level=debug msg="serving 'healthz' on ':7979/healthz'"
time="2025-10-20T14:59:32Z" level=debug msg="serving 'metrics' on ':7979/metrics'"
time="2025-10-20T14:59:32Z" level=debug msg="registered '19' metrics"
time="2025-10-20T14:59:32Z" level=info msg="Created Kubernetes client https://10.3.0.1:443"
time="2025-10-20T14:59:33Z" level=info msg="OVH: 3 zones found"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting records for domain_b.com from API"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting records for domain_a.com from API"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting records for domain_c.com from API"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578359 for domain_a.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578356 for domain_a.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578355 for domain_a.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578357 for domain_a.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578358 for domain_a.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578478 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578477 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578479 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578480 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578481 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578526 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578482 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578525 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578527 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578528 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386578529 for domain_b.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386185734 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386186259 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386185735 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386185736 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386185742 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386185741 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386186256 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386186258 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Getting record 5386185740 for domain_c.com"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578359 for domain_a.com is record#5386578359: NS |  => dns16.ovh.net. (0)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578358 for domain_a.com is record#5386578358: NS |  => ns16.ovh.net. (0)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578481 for domain_b.com is record#5386578481: NS |  => dns16.ovh.net. (0)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578526 for domain_b.com is record#5386578526: A | master.prod => 51.91.161.159 (60)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578482 for domain_b.com is record#5386578482: NS |  => ns16.ovh.net. (0)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578525 for domain_b.com is record#5386578525: TXT | myorg-3--prod--a-www => \"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app2--myorg-3--prod--master-live\" (0)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578527 for domain_b.com is record#5386578527: A |  => 51.91.161.149 (60)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578528 for domain_b.com is record#5386578528: A | www => 51.91.161.149 (60)"
time="2025-10-20T14:59:33Z" level=debug msg="OVH: Record 5386578529 for domain_b.com is record#5386578529: TXT | myorg-3--prod--a-master.prod => \"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app2--myorg-3--prod--master\" (0)"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Record 5386185734 for domain_c.com is record#5386185734: NS |  => dns16.ovh.net. (0)"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Record 5386185735 for domain_c.com is record#5386185735: NS |  => ns16.ovh.net. (0)"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Record 5386185736 for domain_c.com is record#5386185736: A |  => 213.186.33.5 (0)"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Record 5386185742 for domain_c.com is record#5386185742: TXT | www => \"3|welcome\" (0)"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Record 5386185741 for domain_c.com is record#5386185741: TXT |  => \"1|www.domain_c.com\" (0)"
time="2025-10-20T14:59:34Z" level=info msg="OVH: 11 endpoints have been found"
time="2025-10-20T14:59:34Z" level=debug msg="Endpoints generated from ingress: default/app1--myorg-3--prod--master-live: [domain_a.com 30 IN A  51.91.161.149 [] domain_a.com 30 IN A  51.91.161.149 [] www.domain_a.com 30 IN A  51.91.161.149 []]"
time="2025-10-20T14:59:34Z" level=debug msg="Endpoints generated from ingress: default/app2--myorg-3--prod--master: [master.prod.domain_b.com 30 IN A  51.91.161.149 [] master.prod.domain_b.com 30 IN A  51.91.161.149 []]"
time="2025-10-20T14:59:34Z" level=debug msg="Endpoints generated from ingress: default/app2--myorg-3--prod--master-live: [domain_b.com 30 IN A  51.91.161.149 [] domain_b.com 30 IN A  51.91.161.149 [] www.domain_b.com 30 IN A  51.91.161.149 []]"
time="2025-10-20T14:59:34Z" level=debug msg="Endpoints generated from ingress: default/app1--myorg-3--prod--master: [master.prod.domain_a.com 30 IN A  51.91.161.149 [] master.prod.domain_a.com 30 IN A  51.91.161.149 []]"
time="2025-10-20T14:59:34Z" level=debug msg="Removing duplicate endpoint domain_a.com 30 IN A  51.91.161.149 []"
time="2025-10-20T14:59:34Z" level=debug msg="Removing duplicate endpoint master.prod.domain_b.com 30 IN A  51.91.161.149 []"
time="2025-10-20T14:59:34Z" level=debug msg="Removing duplicate endpoint domain_b.com 30 IN A  51.91.161.149 []"
time="2025-10-20T14:59:34Z" level=debug msg="Removing duplicate endpoint master.prod.domain_a.com 30 IN A  51.91.161.149 []"
time="2025-10-20T14:59:34Z" level=debug msg="Skipping endpoint domain_b.com 60 IN A  51.91.161.149 [] because owner id does not match, found: \"\", required: \"myorg-3--prod--\""
time="2025-10-20T14:59:34Z" level=debug msg="Skipping endpoint domain_b.com 30 IN A  51.91.161.149 [] because owner id does not match, found: \"\", required: \"myorg-3--prod--\""
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes CREATE dns:\"domain_a.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes CREATE dns:\"www.domain_a.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes CREATE dns:\"master.prod.domain_a.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes CREATE dns:\"myorg-3--prod--a-domain_a.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app1--myorg-3--prod--master-live\" / type:TXT"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes CREATE dns:\"myorg-3--prod--a-www.domain_a.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app1--myorg-3--prod--master-live\" / type:TXT"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes CREATE dns:\"myorg-3--prod--a-master.prod.domain_a.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app1--myorg-3--prod--master\" / type:TXT"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATEOLD dns:\"master.prod.domain_b.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATEOLD dns:\"www.domain_b.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATEOLD dns:\"myorg-3--prod--a-master.prod.domain_b.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app2--myorg-3--prod--master\" / type:TXT"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATEOLD dns:\"myorg-3--prod--a-www.domain_b.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app2--myorg-3--prod--master-live\" / type:TXT"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATENEW dns:\"master.prod.domain_b.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATENEW dns:\"www.domain_b.com\" / targets:51.91.161.149 / type:A"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATENEW dns:\"myorg-3--prod--a-master.prod.domain_b.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app2--myorg-3--prod--master\" / type:TXT"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: changes UPDATENEW dns:\"myorg-3--prod--a-www.domain_b.com\" / targets:\"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app2--myorg-3--prod--master-live\" / type:TXT"
time="2025-10-20T14:59:34Z" level=info msg="OVH: \"\": 6 changes will be done"
time="2025-10-20T14:59:34Z" level=info msg="OVH: \"domain_b.com\": 0 changes will be done"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Refresh domain_b.com zone"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Add an entry to  zone action(create) : myorg-3--prod--a-master.prod.domain_a.com 0 IN TXT \"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app1--myorg-3--prod--master\""
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Add an entry to  zone action(create) : domain_a.com 30 IN A 51.91.161.149"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Add an entry to  zone action(create) : www.domain_a.com 30 IN A 51.91.161.149"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Add an entry to  zone action(create) : master.prod.domain_a.com 30 IN A 51.91.161.149"
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Add an entry to  zone action(create) : myorg-3--prod--a-domain_a.com 0 IN TXT \"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app1--myorg-3--prod--master-live\""
time="2025-10-20T14:59:34Z" level=debug msg="OVH: Add an entry to  zone action(create) : myorg-3--prod--a-www.domain_a.com 0 IN TXT \"heritage=external-dns,external-dns/owner=myorg-3--prod--,external-dns/resource=ingress/default/app1--myorg-3--prod--master-live\""
time="2025-10-20T14:59:34Z" level=error msg="Failed to do run once: soft error\nOVHcloud API error (status code 404): Client::NotFound: \"Got an invalid (or empty) URL\" (X-OVH-Query-Id: EU.ext-3.68f64e56.840806.eae7a155ecaed666ecb3897024503631) (consecutive soft errors: 1)"

Anything else we need to know?:

When I deploy external-dns with version v1.16.1 I don't get this error and records are created.

I think this output might reveal the error: Add an entry to zone action(create). There are two spaces between to and zone. Looks like an empty string is used instead of the zone-id.

Please let me know if you need more information.

I don't know how to further debug this, so reverting back to v1.16.1 for now.

Environment:

  • External-DNS version >= 1.17.0
  • DNS provider: OVHCloud
  • Others:

feluxe avatar Oct 20 '25 15:10 feluxe

I have a question. Maybe this is realted:

My domain contains 'umlauts' and therefore is punycode encoded. E.g. xn--berstuff-55a.de for überstuff.de. I have to use the puny encoded string for the ingresses, like in this example:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "my-app"
  labels:
    app: "my-app"
  annotations:
    kubernetes.io/ingress.class: nginx
    external-dns.alpha.kubernetes.io/ttl: "30"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    external-dns.alpha.kubernetes.io/hostname: "xn--berstuff-55a.de"

spec:

  tls:
  - ...

  rules:

  - host: "xn--berstuff-55a.de"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: "my-app"
            port:
              number: 3000

I'm wondering if this applies to external-dns.alpha.kubernetes.io/hostname as well.

I'm not sure which one I should use here:

  annotations:
    external-dns.alpha.kubernetes.io/hostname: "xn--berstuff-55a.de"
    # or
    external-dns.alpha.kubernetes.io/hostname: "überstuff.de"

I tried with both and neither worked.

feluxe avatar Oct 20 '25 16:10 feluxe

Hard to say, not sure in which form OVH API expects domains with punycode characters.

Which option worked in 0.16.1?

ivankatliarchuk avatar Oct 20 '25 18:10 ivankatliarchuk

From first look external-dns.alpha.kubernetes.io/hostname: "überstuff.de" should be just fine, as the code does normalisation https://github.com/kubernetes-sigs/external-dns/blob/017f7687ca393138b565e51757a6fa8010902066/plan/plan.go#L341, and it could in theory work with normilized form. Worth to isolate error, as not clear what exactly the cause is.

ivankatliarchuk avatar Oct 20 '25 19:10 ivankatliarchuk

Which option worked in 0.16.1?

On 0.16.1 it works with:

external-dns.alpha.kubernetes.io/hostname: "überstuff.de"

The same config doesn't work with versions higher than 0.16.1. I tried various settings to make this work with 0.19.0 without luck.

I have the same problem with all domains that I tried (only one of them contains umlauts).

feluxe avatar Nov 03 '25 20:11 feluxe

I was able to get it working on v0.19.0, but only with "normal" domains (i.e., domains without umlauts or punycode).

Here’s what I had to do:

  • Manually create an A record for the apex domain with any IP, e.g.:
    60 IN A 50.59.218.140
  • Manually create a TXT record to mark ownership:
    my-cluster--prod--a 60 IN TXT "heritage=external-dns,external-dns/owner=my-cluster--prod,external-dns/resource=ingress/default/my-app-ingress-name"

After doing this, the error Got an invalid (or empty) URL... disappeared, and external-dns updated the A record to the correct IP address. All other subdomains were created automatically, without manual intervention.

Here is the config I used:

args:
- --provider=ovh
- --source=ingress
- --policy=sync
- --log-level=trace
- --interval=2m
- --registry=txt
- --txt-owner-id=my-cluster--prod
- --txt-prefix=my-cluster--prod--%{record_type}.
- --txt-cache-interval=10s

However, the same approach does not work with punycode domains. For those, this error remains:

time="2025-10-20T15:04:33Z" level=error msg="Failed to do run once: soft error\nOVHcloud API error (status code 404): Client::NotFound: \"Got an invalid (or empty) URL\" (X-OVH-Query-Id: EU.ext-2.68f64f81.842220.afb3a70c1c04ffdb199cc36e15ad7a9b) (consecutive soft errors: 2)"

It's very frustrating trying to get this to work (if it's even possible) with OVHcloud. I'm writing this in the hope that someone from OVH might address it.

feluxe avatar Nov 04 '25 18:11 feluxe

@rbeuque74 Do you think you can reproduce and investigate this one ?

mloiseleur avatar Nov 09 '25 21:11 mloiseleur

Hello,

I think this output might reveal the error: Add an entry to zone action(create). There are two spaces between to and zone. Looks like an empty string is used instead of the zone-id.

I confirm that the Query ID provided reflect an empty DNS zone ID used inside the API. I will try to play with some punny-code zones to understand the behavior of the ExternalDNS engine.

rbeuque74 avatar Nov 12 '25 16:11 rbeuque74

/assign

rbeuque74 avatar Nov 12 '25 16:11 rbeuque74

Hello @mloiseleur @ivankatliarchuk,

I believe this change #5147 introduced the regression on OVH provider side.

I suggest a change in the behavior of provider/zonefinder.go https://github.com/rbeuque74/external-dns/blob/fix-ovh-issue-5914/provider/zonefinder.go#L29-L35

Currently, record name are transformed to Unicode, while zone name are not transformed, causing the zone to be not found. I suggest that we have an identical process for zone name and record name, all to be changed into Unicode. Otherwise, we should add a check in the Add function to make sure provider are not pushing punicode zone here.

In the mean time, I will rework the way we are treating this function output to make sure it is future proof with better error message.

What do you think? Romain

rbeuque74 avatar Nov 13 '25 17:11 rbeuque74

Hello, I opened a pull-request regarding this issue, to fix the behavior in the core library of external-dns.

You can now watch #5980 for fix

Thanks

rbeuque74 avatar Nov 27 '25 11:11 rbeuque74