Inconsistency between informer cache and etcd

[brucez-anyscale]

I have a problem for my k8s controller.

My controller reconcile logic is:

1. Get the CR
2. If CR does not exist, print a log "not exist". Finish reconcile loop.
3. If CR exists, update the CR state with a counter and print "CR updated". Reconcile again after 100ms with RequeueAfter.
4. If CR exists, and the CR is updated 100 times (check the CR state), delete the CR. Reconcile again after 100ms with RequeueAfter.

I start my reconcile with builder.WithPredicates(predicate.GenerationChangedPredicate{}).

What happened,

1. My reconcile loop deletes the CR.
2. Then I manually create the CR again.
3. My reconcile loop prints a log says "CR updated" only once.
4. My reconcile loop prints "not exist".

How could that happen? Could this because when I update the CR state, it directly writes to etcd. But when I read the CR, it reads from informer cache?