Stale controller runtime cache can confuse the clients including UI
Description
After the UI makes a create/update/delete call, it calls List, only to find that the response doesn't reflect the change that was just made.
My theory is this is caused by a stale controller-runtime cache that doesn't reflect reality after the update.
If true, I'm not sure how to fix this, given that there can be multiple replicas of api-server, and while we may be able to ensure the cache is up to date for the replica handling the update/delete/create, we can't be sure the peer replica also has up-to-date cache.
Screenshots
https://github.com/akuity/kargo/assets/12677113/aa0ef176-b341-411c-844a-e8857a1761bc
Steps to Reproduce
Play with the new Role UI
Version
v0.6.0
Logs
Paste any relevant application logs here.
@jessesuen @rbreeze refresh my memory... did we not end up attributing this odd behavior to something else?
Let's close this. I don't think I've seen this since