cloudflare-operator icon indicating copy to clipboard operation
cloudflare-operator copied to clipboard
verified publisher icon adyanth

cloudflared deployment restarts endlessly when using FluxCD

Open maxpain opened this issue 1 year ago • 9 comments

After a couple of changes in TunnelBinding, the cloudflared deployment started to restarting endlessly.

2024-10-29T11:02:48Z	INFO	Selected protocol	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "protocol": "http"}
2024-10-29T11:02:48Z	INFO	generated cloudflare config	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "hostname": "prerender.example.com", "target": "http://prerender.fastcup.svc:80"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "status": {"hostnames":"prerender.example.com","services":[{"hostname":"prerender.example.com","target":"http://prerender.fastcup.svc:80"}]}}
2024-10-29T11:02:48Z	DEBUG	events	Configuring ConfigMap	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "Configuring"}
2024-10-29T11:02:48Z	DEBUG	events	Applying ConfigMap to Deployment	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "ApplyingConfig"}
2024-10-29T11:02:48Z	DEBUG	events	Applying ConfigMap to Deployment	{"type": "Normal", "object": {"kind":"Deployment","namespace":"cloudflare-operator-system","name":"fastcup","uid":"814b343c-65a3-4220-bb18-1c409761549c","apiVersion":"apps/v1","resourceVersion":"6077156"}, "reason": "ApplyingConfig"}
2024-10-29T11:02:48Z	INFO	Restarted deployment	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0"}
2024-10-29T11:02:48Z	DEBUG	events	ConfigMap applied to Deployment	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "AppliedConfig"}
2024-10-29T11:02:48Z	DEBUG	events	ConfigMap applied to Deployment	{"type": "Normal", "object": {"kind":"Deployment","namespace":"cloudflare-operator-system","name":"fastcup","uid":"814b343c-65a3-4220-bb18-1c409761549c","apiVersion":"apps/v1","resourceVersion":"6077176"}, "reason": "AppliedConfig"}
2024-10-29T11:02:48Z	DEBUG	events	Configured Cloudflare Tunnel	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "Configured"}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "08a32c38-8008-4be1-8496-c7e47f6f629b"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "08a32c38-8008-4be1-8496-c7e47f6f629b"}
2024-10-29T11:02:48Z	DEBUG	events	TunnelBinding Finalizer and Labels added	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "MetaSet"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "08a32c38-8008-4be1-8496-c7e47f6f629b", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "cb1f0236-13f2-481c-a39d-96e1f0f9806e"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "cb1f0236-13f2-481c-a39d-96e1f0f9806e"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "cb1f0236-13f2-481c-a39d-96e1f0f9806e", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "6cf62f1c-ea6d-4003-a8e1-19acb561fb8f"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "6cf62f1c-ea6d-4003-a8e1-19acb561fb8f"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "6cf62f1c-ea6d-4003-a8e1-19acb561fb8f", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "c80cf0d7-0403-4380-979f-b846a36d48ae"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "c80cf0d7-0403-4380-979f-b846a36d48ae"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "c80cf0d7-0403-4380-979f-b846a36d48ae", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "7b1bac63-e396-479f-91f0-a89fe93e27e8"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "7b1bac63-e396-479f-91f0-a89fe93e27e8"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "7b1bac63-e396-479f-91f0-a89fe93e27e8", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:50Z	INFO	Updating existing record	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com", "dnsId": "redacted"}
2024-10-29T11:02:50Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "97105764-b203-4b10-a27e-17ba540c4c7c"}
2024-10-29T11:02:50Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "97105764-b203-4b10-a27e-17ba540c4c7c"}
2024-10-29T11:02:50Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "97105764-b203-4b10-a27e-17ba540c4c7c", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:50Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "1263b978-ca6a-43fb-b7a1-ed4b93896418"}
2024-10-29T11:02:50Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "1263b978-ca6a-43fb-b7a1-ed4b93896418"}
2024-10-29T11:02:50Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "1263b978-ca6a-43fb-b7a1-ed4b93896418", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:50Z	INFO	DNS record updated successfully	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com"}
2024-10-29T11:02:50Z	INFO	Updating existing TXT record	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com", "dnsId": "redacted", "txtId": "redacted"}
2024-10-29T11:02:51Z	INFO	DNS record updated successfully	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com"}
2024-10-29T11:02:51Z	INFO	Inserted/Updated DNS/TXT entry	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0"}
2024-10-29T11:02:51Z	DEBUG	events	Inserted/Updated DNS/TXT entry	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "CreatedDns"}

maxpain avatar Oct 29 '24 11:10 maxpain

The problem only exists if using Flux CD. I have this TunnelBinding config:

apiVersion: networking.cfargotunnel.com/v1alpha1
kind: TunnelBinding
metadata:
  name: prerender
subjects:
  - name: prerender
tunnelRef:
  kind: ClusterTunnel
  name: fastcup

And cloudflare-operator changes subjects, adding spec:

subjects:
  - kind: Service
    name: prerender
    spec:
      noTlsVerify: false
      proxyAddress: 127.0.0.1
      proxyPort: 0
      proxyType: ''

After this, Flux CD reconciles this Custom Resource again.

maxpain avatar Oct 29 '24 11:10 maxpain

Why does cloudflare-operator change this CR at all? Is it necessary?

maxpain avatar Oct 29 '24 11:10 maxpain

The controller adds labels and sets the status on the tunnel binding, which is required to track it. I use this with ArgoCD and it does not seem to have problems with diffs, so this seems to be something you could let Flux ignore?

https://github.com/adyanth/cloudflare-operator/blob/d6135dd4919787cadff60b59e7a63b8b7db49265/controllers/tunnelbinding_controller.go#L250

adyanth avatar Oct 30 '24 15:10 adyanth

@adyanth FluxCD ignores changes in status and labels, but doesn't ignore spec changes. I think changing spec is an antipattern.

maxpain avatar Oct 30 '24 15:10 maxpain

The controller does not explicitly change the values in the spec. I believe those are defaults getting serialized, which might be either a JSON tag I'm missing or I might need to be using pointers for it to be nullable to not be serialized. I am not fully sure since my instance only added the boolean for noTlsVerify and not the rest.

adyanth avatar Oct 30 '24 15:10 adyanth

I had the same issue with ArgoCD and was able to work around it by adding to the Application spec:

  ignoreDifferences:
  - group: networking.cfargotunnel.com
    kind: TunnelBinding
    jsonPointers:
    - /subjects/spec

benperove avatar Nov 02 '24 18:11 benperove

@benperove you should not need to ignore the whole spec, that is surprising.

adyanth avatar Nov 05 '24 09:11 adyanth

Do you see this problem persist in v0.13.0?

adyanth avatar May 06 '25 06:05 adyanth

Yes, the problem persists for me using FluxCD paired with cloudflare-operator. I have reconcilation set at 1 min, which updates the records on Cloudflare every minute as well.

My current workaround is to add a fully qualified spec, similar to:

spec:
  http2Origin: false
  noTlsVerify: false
  proxyAddress: 127.0.0.1
  proxyPort: 0
  proxyType: ""

That way the record doesn't get updated, and Cloudflare doesn't get overloaded by requests. It would be great if this wasn't necessary though

jvdwrf avatar Jun 08 '25 02:06 jvdwrf