argo-rollouts icon indicating copy to clipboard operation
argo-rollouts copied to clipboard

Published 20 hours ago • verified publisher icon argoproj

30 second pause between replica pod creation canary strategy

Open gradyela opened this issue 2 years ago • 3 comments

Summary

What happened/what you expected to happen?

When applying a rollout change, I'd expect to see replica pods spin up simultaneously based on the step settings. Currently, replica pods spin up every 30 seconds in exponential steps. Ex: (ignore the first two pods as this was just a scale up event, but you can see that the two pods didn't start spinning up until 30 seconds after the revision.

NAME                                          KIND        STATUS        AGE    INFO
⟳ fraud-monitoring                            Rollout     ✔ Healthy     76d    
├──# revision:6                                                                
│  └──⧉ fraud-monitoring-7c948999fb           ReplicaSet  ✔ Healthy     3m3s   stable
│     ├──□ fraud-monitoring-7c948999fb-6fjgx  Pod         ✔ Running     2m33s  ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-h64pj  Pod         ✔ Running     2m3s   ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-qwbvn  Pod         ✔ Running     2m3s   ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-8kc6v  Pod         ✔ Running     93s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-9t94t  Pod         ✔ Running     93s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-blmph  Pod         ✔ Running     93s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-cr6qn  Pod         ✔ Running     93s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-88m9v  Pod         ✔ Running     63s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-8bcp9  Pod         ✔ Running     63s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-fhphx  Pod         ✔ Running     63s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-fqblx  Pod         ✔ Running     63s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-hz4qm  Pod         ✔ Running     63s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-k2jh7  Pod         ✔ Running     63s    ready:1/1
│     ├──□ fraud-monitoring-7c948999fb-nl5ps  Pod         ✔ Running     63s    ready:1/1
│     └──□ fraud-monitoring-7c948999fb-tn6zt  Pod         ✔ Running     63s    ready:1/1

Diagnostics

What version of Argo Rollouts are you running? 1.20-rc.2

# Paste the logs from the rollout controller

# Logs for the entire controller:
time="2022-05-13T01:12:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=15 namespace=dev resourceVersion=37408287 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Pod template change detected (new: 7c948999fb, old: 549fb6bb55)" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Assuming 7c948999fb for new replicaset pod hash" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Pod template change detected (new: 7c948999fb, old: 549fb6bb55)" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:12:48Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"Rollout \\\"fraud-monitoring\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"currentPodHash\":\"7c948999fb\",\"message\":\"more replicas need to be updated\",\"observedGeneration\":\"15\",\"phase\":\"Progressing\",\"updatedReplicas\":null}}" generation=15 namespace=dev resourceVersion=37408287 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="persisted to informer" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=15 namespace=dev resourceVersion=37408287 rollout=fraud-monitoring time_ms=19.932327
time="2022-05-13T01:12:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Updating replica set 'fraud-monitoring-7c948999fb' revision from 0 to 6" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Created ReplicaSet fraud-monitoring-7c948999fb" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:12:48Z" level=info msg="Rollout updated to revision 6" event_reason=RolloutUpdated namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Created ReplicaSet fraud-monitoring-7c948999fb (revision 6)" event_reason=NewReplicaSetCreated namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"dev\", Name:\"fraud-monitoring\", UID:\"4bda1b6e-954b-4b2f-85c7-e29b39b1ef1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"37408291\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 6"
time="2022-05-13T01:12:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"dev\", Name:\"fraud-monitoring\", UID:\"4bda1b6e-954b-4b2f-85c7-e29b39b1ef1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"37408291\", FieldPath:\"\"}): type: 'Normal' reason: 'NewReplicaSetCreated' Created ReplicaSet fraud-monitoring-7c948999fb (revision 6)"
time="2022-05-13T01:12:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:12:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Set rollout condition: &RolloutCondition{Type:Progressing,Status:True,LastUpdateTime:2022-05-13 01:12:48.418794705 +0000 UTC m=+1725533.366929024,LastTransitionTime:2022-05-13 01:12:48.418794802 +0000 UTC m=+1725533.366929116,Reason:NewReplicaSetCreated,Message:Created new replica set \"fraud-monitoring-7c948999fb\",}" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring time_ms=47.236056999999995
time="2022-05-13T01:12:48Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="rollout syncHandler queue retries: 6 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:12:48.439872       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Scaled up ReplicaSet fraud-monitoring-7c948999fb (revision 6) from 0 to 15" event_reason=ScalingReplicaSet namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Not finished reconciling new ReplicaSet 'fraud-monitoring-7c948999fb'" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Not finished reconciling ReplicaSets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:12:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"dev\", Name:\"fraud-monitoring\", UID:\"4bda1b6e-954b-4b2f-85c7-e29b39b1ef1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"37408294\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet fraud-monitoring-7c948999fb (revision 6) from 0 to 15"
time="2022-05-13T01:12:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:12:48 +0000 UTC - now: 2022-05-13 01:12:48.46230173 +0000 UTC m=+1725533.410436036]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Patched: {\"status\":{\"observedGeneration\":\"16\"}}" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring time_ms=36.005694
time="2022-05-13T01:12:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:12:48 +0000 UTC - now: 2022-05-13 01:12:48.49074388 +0000 UTC m=+1725533.438878188]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=15.038164
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:12:48 +0000 UTC - now: 2022-05-13 01:12:48.508190117 +0000 UTC m=+1725533.456324437]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=17.430065000000003
time="2022-05-13T01:14:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:48Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=39.291872999999995
time="2022-05-13T01:14:48Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="rollout syncHandler queue retries: 3 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:14:48.653237       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-549fb6bb55: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:48Z" level=info msg="Scaled down ReplicaSet fraud-monitoring-549fb6bb55 (revision 5) from 15 to 8" event_reason=ScalingReplicaSet namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Not finished reconciling stableRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Not finished reconciling ReplicaSets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"dev\", Name:\"fraud-monitoring\", UID:\"4bda1b6e-954b-4b2f-85c7-e29b39b1ef1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"37408299\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet fraud-monitoring-549fb6bb55 (revision 5) from 15 to 8"
time="2022-05-13T01:14:48Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":30,\"availableReplicas\":22,\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:12:48Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:14:48Z\",\"message\":\"ReplicaSet \\\"fraud-monitoring-7c948999fb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"message\":\"old replicas are pending termination\",\"readyReplicas\":22,\"replicas\":30,\"updatedReplicas\":15}}" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=66.500664
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-549fb6bb55: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:48 +0000 UTC - now: 2022-05-13 01:14:48.742640875 +0000 UTC m=+1725653.690775181]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring time_ms=23.093948
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-549fb6bb55: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:48 +0000 UTC - now: 2022-05-13 01:14:48.759660849 +0000 UTC m=+1725653.707795174]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring time_ms=16.993821999999998
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":23,\"availableReplicas\":15,\"readyReplicas\":15,\"replicas\":23}}" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring time_ms=28.370297
time="2022-05-13T01:14:48Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:48 +0000 UTC - now: 2022-05-13 01:14:48.809631739 +0000 UTC m=+1725653.757766045]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=16.756238
time="2022-05-13T01:14:58Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=21.713621
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 3 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:14:58.770453       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=24.522921
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 5 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:14:58.795103       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=22.810897
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 7 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:14:58.818084       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=6.168013
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 9 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:14:58.824332       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-7c948999fb: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=8.361923
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 11 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
E0513 01:14:58.847941       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "fraud-monitoring-7c948999fb": the object has been modified; please apply your changes to the latest version and try again
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Scaled down ReplicaSet fraud-monitoring-549fb6bb55 (revision 5) from 8 to 0" event_reason=ScalingReplicaSet namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Not finished reconciling stableRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Not finished reconciling ReplicaSets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"dev\", Name:\"fraud-monitoring\", UID:\"4bda1b6e-954b-4b2f-85c7-e29b39b1ef1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"37409178\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet fraud-monitoring-549fb6bb55 (revision 5) from 8 to 0"
time="2022-05-13T01:14:59Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-549fb6bb55: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:59Z" level=info msg="Rollout completed update to revision 6 (7c948999fb): Completed all 0 canary steps" event_reason=RolloutCompleted namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"dev\", Name:\"fraud-monitoring\", UID:\"4bda1b6e-954b-4b2f-85c7-e29b39b1ef1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"37409178\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutCompleted' Rollout completed update to revision 6 (7c948999fb): Completed all 0 canary steps"
time="2022-05-13T01:14:59Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":23,\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:12:48Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:14:59Z\",\"message\":\"ReplicaSet \\\"fraud-monitoring-7c948999fb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"readyReplicas\":23,\"stableRS\":\"7c948999fb\"}}" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=47.973808999999996
time="2022-05-13T01:14:59Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Processing completed" resource=dev/fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Cleaning up 3 old replicasets from revision history limit 2" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Looking to cleanup old replica sets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Trying to cleanup replica set \"fraud-monitoring-5874fdffc\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-549fb6bb55: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:59 +0000 UTC - now: 2022-05-13 01:14:59.432323735 +0000 UTC m=+1725664.380458041]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=23.351269000000002
time="2022-05-13T01:14:59Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-5874fdffc: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:59 +0000 UTC - now: 2022-05-13 01:14:59.433945029 +0000 UTC m=+1725664.382079337]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=1.5497530000000002
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:59 +0000 UTC - now: 2022-05-13 01:14:59.43528089 +0000 UTC m=+1725664.383415195]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=1.300718
time="2022-05-13T01:14:59Z" level=info msg="Enqueueing parent of dev/fraud-monitoring-549fb6bb55: Rollout dev/fraud-monitoring"
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":15,\"availableReplicas\":15,\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:14:59Z\",\"lastUpdateTime\":\"2022-05-13T01:14:59Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"True\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:14:59Z\",\"message\":\"ReplicaSet \\\"fraud-monitoring-7c948999fb\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"message\":null,\"phase\":\"Healthy\",\"readyReplicas\":15,\"replicas\":15}}" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409355 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=25.613796
time="2022-05-13T01:14:59Z" level=info msg="Start processing" resource=dev/fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Processing completed" resource=dev/fraud-monitoring

# Logs for a specific rollout:
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=15 namespace=dev resourceVersion=37408287 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Pod template change detected (new: 7c948999fb, old: 549fb6bb55)" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Assuming 7c948999fb for new replicaset pod hash" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Pod template change detected (new: 7c948999fb, old: 549fb6bb55)" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:12:48Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"Rollout \\\"fraud-monitoring\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"currentPodHash\":\"7c948999fb\",\"message\":\"more replicas need to be updated\",\"observedGeneration\":\"15\",\"phase\":\"Progressing\",\"updatedReplicas\":null}}" generation=15 namespace=dev resourceVersion=37408287 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="persisted to informer" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=15 namespace=dev resourceVersion=37408287 rollout=fraud-monitoring time_ms=19.932327
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Updating replica set 'fraud-monitoring-7c948999fb' revision from 0 to 6" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Created ReplicaSet fraud-monitoring-7c948999fb" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Rollout updated to revision 6" event_reason=RolloutUpdated namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Created ReplicaSet fraud-monitoring-7c948999fb (revision 6)" event_reason=NewReplicaSetCreated namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Set rollout condition: &RolloutCondition{Type:Progressing,Status:True,LastUpdateTime:2022-05-13 01:12:48.418794705 +0000 UTC m=+1725533.366929024,LastTransitionTime:2022-05-13 01:12:48.418794802 +0000 UTC m=+1725533.366929116,Reason:NewReplicaSetCreated,Message:Created new replica set \"fraud-monitoring-7c948999fb\",}" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=15 namespace=dev resourceVersion=37408288 rollout=fraud-monitoring time_ms=47.236056999999995
time="2022-05-13T01:12:48Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="rollout syncHandler queue retries: 6 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Scaled up ReplicaSet fraud-monitoring-7c948999fb (revision 6) from 0 to 15" event_reason=ScalingReplicaSet namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Not finished reconciling new ReplicaSet 'fraud-monitoring-7c948999fb'" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Not finished reconciling ReplicaSets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:12:48 +0000 UTC - now: 2022-05-13 01:12:48.46230173 +0000 UTC m=+1725533.410436036]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Patched: {\"status\":{\"observedGeneration\":\"16\"}}" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408294 rollout=fraud-monitoring time_ms=36.005694
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:12:48 +0000 UTC - now: 2022-05-13 01:12:48.49074388 +0000 UTC m=+1725533.438878188]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=15.038164
time="2022-05-13T01:12:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:12:48 +0000 UTC - now: 2022-05-13 01:12:48.508190117 +0000 UTC m=+1725533.456324437]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:12:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=17.430065000000003
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=39.291872999999995
time="2022-05-13T01:14:48Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="rollout syncHandler queue retries: 3 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Scaled down ReplicaSet fraud-monitoring-549fb6bb55 (revision 5) from 15 to 8" event_reason=ScalingReplicaSet namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Not finished reconciling stableRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Not finished reconciling ReplicaSets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":30,\"availableReplicas\":22,\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:12:48Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:14:48Z\",\"message\":\"ReplicaSet \\\"fraud-monitoring-7c948999fb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"message\":\"old replicas are pending termination\",\"readyReplicas\":22,\"replicas\":30,\"updatedReplicas\":15}}" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37408299 rollout=fraud-monitoring time_ms=66.500664
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:48 +0000 UTC - now: 2022-05-13 01:14:48.742640875 +0000 UTC m=+1725653.690775181]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring time_ms=23.093948
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:48 +0000 UTC - now: 2022-05-13 01:14:48.759660849 +0000 UTC m=+1725653.707795174]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring time_ms=16.993821999999998
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":23,\"availableReplicas\":15,\"readyReplicas\":15,\"replicas\":23}}" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409154 rollout=fraud-monitoring time_ms=28.370297
time="2022-05-13T01:14:48Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:48 +0000 UTC - now: 2022-05-13 01:14:48.809631739 +0000 UTC m=+1725653.757766045]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:48Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=16.756238
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=21.713621
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 3 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=24.522921
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 5 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=22.810897
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 7 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=6.168013
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 9 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=8.361923
time="2022-05-13T01:14:58Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"fraud-monitoring-7c948999fb\": the object has been modified; please apply your changes to the latest version and try again" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:58Z" level=info msg="rollout syncHandler queue retries: 11 : key \"dev/fraud-monitoring\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Scaled down ReplicaSet fraud-monitoring-549fb6bb55 (revision 5) from 8 to 0" event_reason=ScalingReplicaSet namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Not finished reconciling stableRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Not finished reconciling ReplicaSets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout completed update to revision 6 (7c948999fb): Completed all 0 canary steps" event_reason=RolloutCompleted namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":23,\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:12:48Z\",\"lastUpdateTime\":\"2022-05-13T01:12:48Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:14:59Z\",\"message\":\"ReplicaSet \\\"fraud-monitoring-7c948999fb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"readyReplicas\":23,\"stableRS\":\"7c948999fb\"}}" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409178 rollout=fraud-monitoring time_ms=47.973808999999996
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Cleaning up 3 old replicasets from revision history limit 2" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Looking to cleanup old replica sets" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Trying to cleanup replica set \"fraud-monitoring-5874fdffc\"" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:59 +0000 UTC - now: 2022-05-13 01:14:59.432323735 +0000 UTC m=+1725664.380458041]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=23.351269000000002
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:59 +0000 UTC - now: 2022-05-13 01:14:59.433945029 +0000 UTC m=+1725664.382079337]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=1.5497530000000002
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Timed out (false) [last progress check: 2022-05-13 01:14:59 +0000 UTC - now: 2022-05-13 01:14:59.43528089 +0000 UTC m=+1725664.383415195]" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=1.300718
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":15,\"availableReplicas\":15,\"conditions\":[{\"lastTransitionTime\":\"2022-05-13T01:01:30Z\",\"lastUpdateTime\":\"2022-05-13T01:01:30Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-05-13T01:14:59Z\",\"lastUpdateTime\":\"2022-05-13T01:14:59Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"True\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-02-26T00:31:40Z\",\"lastUpdateTime\":\"2022-05-13T01:14:59Z\",\"message\":\"ReplicaSet \\\"fraud-monitoring-7c948999fb\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"message\":null,\"phase\":\"Healthy\",\"readyReplicas\":15,\"replicas\":15}}" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="persisted to informer" generation=16 namespace=dev resourceVersion=37409355 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409333 rollout=fraud-monitoring time_ms=25.613796
time="2022-05-13T01:14:59Z" level=info msg="Started syncing rollout" generation=16 namespace=dev resourceVersion=37409355 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No TrafficRouting Reconcilers found" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No StableRS exists to reconcile or matches newRS" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Rollout does not have any steps" namespace=dev rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="No status changes. Skipping patch" generation=16 namespace=dev resourceVersion=37409355 rollout=fraud-monitoring
time="2022-05-13T01:14:59Z" level=info msg="Reconciliation completed" generation=16 namespace=dev resourceVersion=37409355 rollout=fraud-monitoring time_ms=1.303261

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

gradyela avatar May 13 '22 01:05 gradyela

Could you also provide kubectl get events for the namespace this is running in. Rollouts just creates the ReplicaSet and the native ReplicaSet controller would generally manage the pod creation. I want to make sure for instance that your cluster did not have to scale in order to add the pods.

zachaller avatar May 13 '22 02:05 zachaller

@zachaller Get events in the namespace that the rollout lives in? I'll need to re-do the rollout, it looks like the events are gone from that specific action. But, looking at the cluster config, its not permitted to scale up nodes right now. Its at its cap.

gradyela avatar May 13 '22 18:05 gradyela

Also, if its helpful, this cluster is running in AWS EKS

gradyela avatar May 13 '22 18:05 gradyela

This issue is stale because it has awaiting-response label for 5 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Dec 07 '22 02:12 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity and has the awaiting-response label.

github-actions[bot] avatar Dec 13 '22 02:12 github-actions[bot]