argo-rollouts icon indicating copy to clipboard operation
argo-rollouts copied to clipboard

Published 20 hours ago • verified publisher icon argoproj

Infinite Ingress recreation when using rancher managed cluster

Open stfnbstl opened this issue 3 years ago • 8 comments

Summary

What happened/what you expected to happen?

I created a canary rollout using nginx for traffic routing. Argo is then creating a second ingress for the canary traffic. Rancher is by default setting some annotations to ingress ressources after creation. As soon as rancher sets the annotations to the canary ingress, argo is removing them because they don't match the desired state that argo knows about. This results in an infinite loop between rancher setting annotations and argo removing them.

This is the annotation rancher sets with sensitive information removed: annotations: field.cattle.io/publicEndpoints: >- [{"addresses":["xxx"],"port":80,"protocol":"HTTP","serviceName":"rollout-demo-dev:demo-canary","ingressName":"rollout-demo-dev:demo-demo-ing-canary","hostname":"xxx","path":"/demo-dev(/|$)(.*)","allNodes":true}]

Diagnostics

What version of Argo Rollouts are you running? 0.10.2

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

time="2021-04-07T13:02:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"rollout-demo-dev\", Name:\"demo\", UID:\"99b773f9-cade-4762-92a9-302c96fcceaf\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"54013056\", FieldPath:\"\"}): type: 'Normal' reason: 'PatchingCanaryIngress' Updating Ingress `demo-demo-ing-canary` to desiredWeight '0'"
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=139.913028
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.661614577 +0000 UTC m=+1976182.027993995)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No changes to canary ingress - skipping patch" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=1.914315
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.710524963 +0000 UTC m=+1976182.076904381)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="updating canary Ingress" desiredWeight=0 ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"rollout-demo-dev\", Name:\"demo\", UID:\"99b773f9-cade-4762-92a9-302c96fcceaf\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"54013056\", FieldPath:\"\"}): type: 'Normal' reason: 'PatchingCanaryIngress' Updating Ingress `demo-demo-ing-canary` to desiredWeight '0'"

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME>

time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=138.819268
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.060928614 +0000 UTC m=+1976181.427308032)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No changes to canary ingress - skipping patch" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=2.004805
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.113631786 +0000 UTC m=+1976181.480011201)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="updating canary Ingress" desiredWeight=0 ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=136.024432
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.259454854 +0000 UTC m=+1976181.625834283)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No changes to canary ingress - skipping patch" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=1.774592
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.308628213 +0000 UTC m=+1976181.675007627)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="updating canary Ingress" desiredWeight=0 ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=142.635665
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.461116764 +0000 UTC m=+1976181.827496173)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No changes to canary ingress - skipping patch" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=1.9575299999999998
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.511738076 +0000 UTC m=+1976181.878117623)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="updating canary Ingress" desiredWeight=0 ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=139.913028
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.661614577 +0000 UTC m=+1976182.027993995)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No changes to canary ingress - skipping patch" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=1.914315
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.710524963 +0000 UTC m=+1976182.076904381)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="updating canary Ingress" desiredWeight=0 ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=143.81683900000002
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.863980903 +0000 UTC m=+1976182.230360313)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No changes to canary ingress - skipping patch" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling StableRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling new ReplicaSet 'demo-677464578c'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling old replica sets" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No Steps remain in the canary steps" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Rollout has executed every step" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="New RS has successfully progressed" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="No status changes. Skipping patch" generation=2 namespace=rollout-demo-dev resourceVersion=54013056 rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciliation completed" namespace=rollout-demo-dev rollout=demo time_ms=2.02496
time="2021-04-07T13:02:34Z" level=info msg="Started syncing rollout at (2021-04-07 13:02:34.914650989 +0000 UTC m=+1976182.281030407)" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="Reconciling TrafficRouting with type 'Nginx'" namespace=rollout-demo-dev rollout=demo
time="2021-04-07T13:02:34Z" level=info msg="updating canary Ingress" desiredWeight=0 ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

stfnbstl avatar Apr 07 '21 13:04 stfnbstl

Rollout does not account for other controllers also managing the Ingress which it created, so it currently syncs all annotations from the reference ingress, to the canary ingress.

The fix would need to be that it preserves injected ingresses from other controllers. This logic would need to go here:

https://github.com/argoproj/argo-rollouts/blob/master/rollout/trafficrouting/nginx/nginx.go#L58-L121

However, the tricky part is that the fix would also have to accommodate the possibility that the reference Ingress had removed annotations, and so the rollout would need to remove only the annotation no longer in the reference ingress, but also preserve the rancher managed annotation.

jessesuen avatar Apr 07 '21 20:04 jessesuen

@stfnbstl can we work around this issue by explicitly setting rancher annotations under additionalIngressAnnotations?

      trafficRouting:
        nginx:
          additionalIngressAnnotations: 
            rancher-annotation: xxxxxxxxx

jessesuen avatar Apr 07 '21 20:04 jessesuen

@jessesuen I tried that because I thought that would be enough for argo to be aware of this annotation and won't overwrite it when the value changed. But with the additional annotations alone, the default prefix combined with the rancher tag is generating an error:

time="2021-04-08T06:02:18Z" level=error msg="error creating canary ingress" err="Ingress.extensions \"demo-demo-ing-canary\" is invalid: metadata.annotations: Invalid value: \"nginx.ingress.kubernetes.io/field.cattle.io/publicEndpoints\": a qualified name must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]') with an optional DNS subdomain prefix and '/' (e.g. 'example.com/MyName')" ingress=demo-demo-ing-canary namespace=rollout-demo-dev rollout=demo

So I changed my rollout configuration to this:

...
strategy:
    canary:
      canaryService: demo-canary
      stableService: demo-stable
      trafficRouting:
        nginx:
          stableIngress: demo-ing
          annotationPrefix: field.cattle.io
          additionalIngressAnnotations:
            publicEndpoints: something
      analysis:
        templates:
        - templateName: demo-success-rate
        startingStep: 2
...

But this results in this error which i cannot really explain (I removed the hostname). I guess it's too much, to change the annotationPrefix for everything.

time="2021-04-08T06:06:54Z" level=error msg="rollout syncHandler error: error creating canary ingress `demo-demo-ing-canary`: admission webhook \"validate.nginx.ingress.kubernetes.io\" denied the request: host \"<hostname>\" and path \"/demo-dev(/|$)(.*)\" is already defined in ingress rollout-demo-dev/demo-ing" namespace=rollout-demo-dev rollout=demo

stfnbstl avatar Apr 08 '21 06:04 stfnbstl

nginx.ingress.kubernetes.io/field.cattle.io/publicEndpoints

Ah yes, I forgot additionalIngressAnnotations also would also incorporate the prefix. So my workaround will not work.

I think we have to go with a fix to preserve injected annotations by other controllers, like rancher

jessesuen avatar Apr 08 '21 23:04 jessesuen

I don't know if it helps but this issue only occurs in combination with NGINX so far. I tried Istio for canary releases and it doesn't seem to be a problem there because there is no clone that argo and rancher are trying to "correct"

stfnbstl avatar Apr 12 '21 05:04 stfnbstl

I'm facing the same issue now after upgrading k8s (rke2) from 1.24.4 to 1.24.8 and upgrading argocd to the current 2.5. I run argo-rollout 1.2 for a while and haven't changed anything else.

I have no clue why it worked before but i'm curious if there is now a workaround?

sigi-tw avatar Dec 05 '22 18:12 sigi-tw

Hello. Who can help with this? I want to suggest a solution. So, what if we add list of ignored annotations? Like this

type NginxTrafficRouting struct {
        ...
	// IgnoredAnnotations list that contains annotations which ignored when compare canary and stable ingress
	// +optional
	IgnoredAnnotations []string `json:"ignoredAnnotations,omitempty" protobuf:"bytes,4,rep,name=ignoredAnnotations"`
}

And use it when create a patch...

patch, modified, err := ingressutil.BuildIngressPatch(
    canaryIngress.Mode(), canaryIngress, desiredCanaryIngress,
    ingressutil.WithAnnotations(r.cfg.Rollout.Spec.Strategy.Canary.TrafficRouting.Nginx.IgnoredAnnotations...),
    ingressutil.WithLabels(), ingressutil.WithSpec(),
)

Update build ingress patch

type patchConfig struct {
	withAnnotations    bool
	excludeAnnotations []string
	withLabels         bool
	withSpec           bool
}

func WithAnnotations(excludeAnnotations ...string) PatchOption {
	return func(p *patchConfig) {
		p.withAnnotations = true
		p.excludeAnnotations = excludeAnnotations
	}
}

func buildIngressPatch(current, desired *networkingv1.Ingress, cfg *patchConfig) ([]byte, bool, error) {
	cur := &networkingv1.Ingress{}
	des := &networkingv1.Ingress{}
	if cfg.withAnnotations {
		cur.Annotations = current.Annotations
		des.Annotations = desired.Annotations

		if len(cfg.excludeAnnotations) > 0 {
			for _, a := range cfg.excludeAnnotations {
				delete(cur.Annotations, a)
				delete(des.Annotations, a)
			}
		}
	}
	if cfg.withLabels {
		cur.Labels = current.Labels
		des.Labels = desired.Labels
	}
	if cfg.withSpec {
		cur.Spec = current.Spec
		des.Spec = desired.Spec
	}
	return diff.CreateTwoWayMergePatch(cur, des, networkingv1.Ingress{})
}

If it's ok then I will create PR...

SergeyChupin avatar Feb 27 '23 19:02 SergeyChupin

Is anyone here? it is still a bug!

letuanvu59 avatar Dec 08 '23 09:12 letuanvu59