error getting cached app managed resources: cache: key is missing

[tooptoop4]

Describe the bug

in the UI i click on a pod, then popup with pod details shows an error 'cache: key is missing' and it doesn't show the pod details. i found that if i click refresh button on the app it does not help, getting same error when clicking on pod. but if i click sync button on app then going to the pod works.

To Reproduce

been using argo for over year and first time this has happened....

Version 2.6.12

Logs

time="2023-07-13T22:27:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=376.227 span.kind=server system=grpc
time="2023-07-13T22:27:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=216.462 span.kind=server system=grpc
time="2023-07-13T22:27:53Z" level=info msg="received unary call /application.ApplicationService/RevisionMetadata" grpc.method=RevisionMetadata grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" revision:\"myrevis\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" span.kind=server system=grpc
time="2023-07-13T22:27:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=RevisionMetadata grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=119.744 span.kind=server system=grpc
time="2023-07-13T22:27:56Z" level=info msg="received unary call /application.ApplicationService/ManagedResources" grpc.method=ManagedResources grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="applicationName:\"myapp\" namespace:\"auth\" name:\"mypod\" kind:\"Pod\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:56Z" span.kind=server system=grpc
time="2023-07-13T22:27:56Z" level=info msg="Requested app 'myapp' refresh"
time="2023-07-13T22:27:57Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" grpc.time_ms=43.44 span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=error msg="finished unary call with code Unknown" error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:56Z" grpc.time_ms=371.861 span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" grpc.time_ms=69.178 span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="received unary call /application.ApplicationService/ManagedResources" grpc.method=ManagedResources grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="applicationName:\"myapp\" namespace:\"auth\" name:\"mypod\" kind:\"Pod\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="Requested app 'myapp' refresh"
time="2023-07-13T22:27:59Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" grpc.time_ms=9.584 span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=error msg="finished unary call with code Unknown" error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" grpc.time_ms=305.273 span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" grpc.time_ms=16.361 span.kind=server system=grpc
time="2023-07-13T22:28:05Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=12350.736 span.kind=server system=grpc

[wilkko]

After upgrading from ArgoCD 2.7.14 to 2.3.8 (to Openshift GitOps operator 1.10) started getting this same error when clicking on certain pods in the UI or when running argocd app diff on command line:

$ argocd app diff autoscale  --local config/autoscale/itest                  
ERRO[0001] finished unary call with code Unknown  error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-10-03T12:06:14+03:00" grpc.time_ms=562.284 span.kind=server system=grpc
FATA[0001] rpc error: code = Unknown desc = error getting cached app managed resources: cache: key is missing

Redis pod has similar looking key for it:

$ redis-cli --scan --pattern 'app|managed-resources|autoscale*'
app|managed-resources|autoscale|1.8.3.gz

[wilkko]

Using argocd login <server> --sso makes the ArgoCD CLI working again.

Issue can be reproduced with:

- oc login... (updates kubekonfig)
- argocd login --core

[jblsk]

We see a variant of this issue with ArgoCD v2.10.1 in argocd-application-controller log with recurring errors like: time="2024-03-05T14:00:11Z" level=error msg="DiffFromCache error: error getting managed resources for app <app_name>: cache: key is missing"

[0xDones]

I eventually get some of those errors on the logs, for every application, even when no one is using the UI or CLI. I wonder if in my case those are really errors or if they should be treated differently, because it may be triggering some false positive alerts.

time="2024-03-19T16:47:00Z" level=error msg="DiffFromCache error: error getting managed resources for app karpenter-dev: cache: key is missing"

ArgoCD version: v2.10.1

[shm-eboks]

seeing these errors pop up as well now

DiffFromCache error: error getting managed resources for app <xxx>: cache: key is missing

running argocd v2.11.4+e1284e1

[omfurman-ma]

We also experience this using:

argocd login --core --sso
 - > Context 'kubernetes' updated
 
argocd app diff .....
-> ERRO[0007] finished unary call with code Unknown         error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService  span.kind=server system=grpc
FATA[0007] rpc error: code = Unknown desc = error getting cached app managed resources: cache: key is missing

[AllanMNoguera]

argocd app sync works good only argocd app diff is having this issue when using --core

time="2024-07-23T15:19:18Z" level=error msg="finished unary call with code Unknown" error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2024-07-23T15:19:14Z" grpc.time_ms=3983.76 span.kind=server system=grpc
time="2024-07-23T15:19:18Z" level=fatal msg="rpc error: code = Unknown desc = error getting cached app managed resources: cache: key is missing"