Anton Gilgur

Hmmm interesting find! > `if event.Object == nil {` Didn't you say that `event` _itself_ was nil? > Maybe this [issue](https://github.com/kubernetes/client-go/issues/334) helps explain this behavior (I also found other complaints...

> @agilgur5 I'm testing the code below, so far everything is fine, but I'm not sure if it's the best approach: A bit hard to read as there isn't syntax...

Not sure about the exact error message, but maybe the resync period duration of `0` from your previous [code comment](https://github.com/argoproj/argo-workflows/issues/11657#issuecomment-1715378071) is too short? I think most of the existing informers...

> We also just updated to 3.4.10 and see this problem as well. > Should we downgrade to 3.4.8? @roitalpaz this seems to have existed well before 3.4.10. as I...

> Maybe it's in general a good idea to change the log level to Debug Log levels should be set based on their context. This is an error and so...

> It seems this is only happening with kubernetes version 1.27 or later. Does anyone have evidence of it happening on earlier versions? 👀 I was wondering why we suddenly...

> maybe this k8s 1.27 change is related? [kubernetes/kubernetes#115620](https://github.com/kubernetes/kubernetes/pull/115620) that seems to be an Informer change, whereas the current code uses a `RetryWatcher` (it is being updated to an Informer...

> can this be treated as a transient error and autoretried by the controller? You can set the [env var](https://argo-workflows.readthedocs.io/en/release-3.4/environment-variables/) `TRANSIENT_ERROR_PATTERN` to add additional patterns to be treated as transient....

> What we have in this case is a successful call to create a pod in the cluster. The pod has been created, in that the kubernetes object is there....

Was the node marked as failed? If not, might be fixed by #12197. This is potentially duplicative of #12231 in that case