Ratify distributed caching leads to request processing slowdown

[akashsinghal]

What happened in your environment?

Enabled Ratify with 3 replicas and EXPERIMENTAL_RATIFY_HIGH_AVAILABILITY enabled with dapr + redis.

For very large clusters 2k pods+, we see some requests timeout due to context cancelled errors. The same issue is not seen when the feature flag is disabled but with 3 replicas.

What did you expect to happen?

No response

What version of Kubernetes are you running?

No response

What version of Ratify are you running?

v1.0.0

Anything else you would like to add?

No response

Are you willing to submit PRs to contribute to this bug fix?

• [X] Yes, I am willing to implement it.