Image update intermittently stops working: "object not found"

[Poundex]

Hello,

We are having a problem with our image update automation, specifically that it sporadically stops working. It will do one or two Git commits and then lock up saying "object not found". But it doesn't specify which object cannot be found, and every now and then it will do a successful push.

The only log message in the controller is {"level":"error","ts":"2023-09-21T10:17:33.875Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind" :"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"xxxxx","namespace":"xxxxxx"},"namespace":"xxxxxx","name":"xxxxx","reconcileID":"354 e2231-3c0a-470e-86c2-7a3b07ec3e61","error":"object not found"}

Trying to manually reconcile I see: $ flux reconcile --verbose image update -n xxxxx xxxxx ► annotating ImageUpdateAutomation xxxxx in xxxxx namespace ✔ ImageUpdateAutomation annotated ◎ waiting for ImageUpdateAutomation reconciliation ✗ ImageUpdateAutomation reconciliation failed: 'object not found'

I can see that all the configuration is good. The ImageRepositories have tag lists, the ImagePolicies have their Latest Image set correctly and like I said it does an update every now and then but then goes immediately back to being stuck.

Different deployments are working/not working at the same time. It is not the case that they are either all working or not working, which means the cluster-wide shared objects (such as the GitRepository and credentials etc) are also fine (which they would have to be as like I said the updates do work sometimes).

We did not see this in testing and went ahead and migrated our development environments to Kubernetes and Flux and now many of them are stuck so if anybody could shed any light I would be extremely grateful.

I made sure the image update controller was latest and updated it to 0.36.1