components-contrib icon indicating copy to clipboard operation
components-contrib copied to clipboard

Redis Entra Id connection stops working after 24hours

Open skbar50 opened this issue 1 year ago • 12 comments

Expected Behavior

We started using Entra Id authentication to Redis with the dapr version 1.14.1. We are able to establish the connection from the AKS pod to Redis using the managed identity used by the pod. The pod runs for many days without restarting. It should be able to connect the Redis whenever there is a request from client.

Actual Behavior

However, after running for about 24hours the pod is failing to connect Redis with the error 'State operation failed: the Dapr endpoint indicated a failure. See InnerException for details. Status(StatusCode="Internal", Detail="fail to get PComSearchCacheProduct from state store statestore: ERR WRONGPASS invalid username-password pair") '

Steps to Reproduce the Problem

Run the pod that connects Redis with Entra ID authentication for more then 24hours. It will fail with the above mentioned error after about 24hours.

Additional information

The AKS pod has been working with the same Redis statestore with the password without any issue. We faced this problem when we tried to implement Entra Id authentication.

skbar50 avatar Sep 25 '24 18:09 skbar50

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Oct 25 '24 19:10 github-actions[bot]

We are still facing the issue. It has not been resolved yet. We need a solution to this.

skbar50 avatar Oct 29 '24 17:10 skbar50

/nostale

yaron2 avatar Oct 30 '24 02:10 yaron2

@yaron2 Any way to escalate this issue? It appears to be a timeout bug in the code and Entra ID auth to Redis fails after 24 hours. Has anyone tested Entra ID auth to Azure Redis after a pod running for 24 hours?

swigerb avatar Nov 20 '24 21:11 swigerb

@swigerb yes this was a bug in the implementation. #3632 would fix this.

shivamkm07 avatar Dec 13 '24 06:12 shivamkm07

The issue is not resolved yet. The AKS pod that runs the redis client application fails after 24 hours. In the log we observed the token is now being refreshed. We suspect the token may not be applied in the connection.

After 24 hours the application is giving this error - 'ERR WRONGPASS invalid username-password pair’

Attached is the application log.

daprlogs.txt

skbar50 avatar Apr 15 '25 21:04 skbar50

Like @skbar50, I ended up here after encountering this exact issue. Initially the Azure Managed Redis setup with Dapr using managed identity works great but stops working after a day.. going back to access keys until this is fixed, which is a bummer.

tswanenberg avatar Aug 25 '25 11:08 tswanenberg

I encountered the same issue. Dapr version: 1.15.9-rc.1

Dapr sidecar logs:

time="2025-09-19T18:11:20.000092785Z" level=info msg="Renewing workload cert; current cert expires on: 2025-09-20 06:18:50 +0000 UTC" app_id=qmmp-dw2merri instance=dw2merri-kubernetes-manager-7559b9cb7-zvbsc scope=dapr.runtime.security type=log ver=1.15.9-rc.1
time="2025-09-19T18:11:20.036528229Z" level=info msg="Successfully renewed workload cert; new cert expires on: 2025-09-20 18:11:20 +0000 UTC" app_id=qmmp-dw2merri instance=dw2merri-kubernetes-manager-7559b9cb7-zvbsc scope=dapr.runtime.security type=log ver=1.15.9-rc.1
time="2025-09-19T18:47:26.546013432Z" level=debug msg="Found Component Outbound Policy for component cache-store: {Timeout:cacheTimeout Retry:cacheRetry CircuitBreaker:cacheCircuitBreaker}" app_id=qmmp-dw2merri instance=dw2merri-kubernetes-manager-7559b9cb7-zvbsc scope=dapr.runtime type=log ver=1.15.9-rc.1
time="2025-09-19T18:47:26.698523315Z" level=warning msg="Error processing operation component[cache-store] output. Retrying in 659.315475ms…" app_id=qmmp-dw2merri instance=dw2merri-kubernetes-manager-7559b9cb7-zvbsc scope=dapr.runtime type=log ver=1.15.9-rc.1
time="2025-09-19T18:47:26.698561815Z" level=debug msg="Error for operation component[cache-store] output was: ERR WRONGPASS invalid username-password pair" app_id=qmmp-dw2merri instance=dw2merri-kubernetes-manager-7559b9cb7-zvbsc scope=dapr.runtime type=log ver=1.15.9-rc.1

As you can see from the logs above, the workload cert was successfully renewed 36 minutes before the request to redis failed. If the pod is restarted, then the requests succeed.

@yaron2 @berndverst Can you please reopen this issue?

georgii-sirotkin avatar Sep 19 '25 18:09 georgii-sirotkin

+1 on this problem on 1.16.1. I was using client credentials which I think has a token that expires after roughly an hour which is exactly what I was seeing. The statestore would work with no issues for approximately an hour before it started failing.

Switched back to the access key for now.

ajstewart avatar Oct 10 '25 14:10 ajstewart

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Nov 09 '25 14:11 github-actions[bot]

Problem is still relevant 12-10 still relevant

tswanenberg avatar Nov 10 '25 07:11 tswanenberg

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Dec 10 '25 07:12 github-actions[bot]