Inconsistency in redis key formation in ArgoCD 2.8.4 with compression enabled

[purajit]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ ] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

ArgoCD 2.8 enables Redis compression by default. However, after upgrading, I found I was unable to run commands like argocd app diff/manifest, since it would fail with

ERRO[0031] finished unary call with code Unknown         error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-10-11T10:30:18-07:00" grpc.time_ms=1554.185 span.kind=server system=grpc

Everything else runs without issue - syncing, UI browsing, etc.

To reproduce Unsure what to mention here.

What I've tried that didn't help:

• verbose logging; but unfortunately nothing relevant shows up
• ensuring REDIS_COMPRESSION is gzip on all pods
• explicitly setting the redis.compression config to gzip
• explicitly setting --redis-compress=gzip as a server argument

What works:

• explicitly setting the redis.compression config to none

Current configuration:

• We use --insecure --enable-gzip (we handle TLS termination ourselves)
• We use the public.ecr.aws/docker/library/haproxy,redis repositories, with HA enabled.

Expected behavior

Everything should "just work", and key formation should be consistent.

Version

We're using the ArgoCD helm chart (v5.46.7).

# on server pod
> /usr/local/bin/argocd-server version
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:12:09Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64

# on local
> argocd version
argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.9.0+g7ceeda6
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

Logs

I cleared out some cache keys, and then ran redis-cli monitor | grep <application> on the Redis pods:

1697046390.058300 [0 172.20.65.196:6379] "SET" "app|managed-resources|<application>|1.8.3.gz"
1697046390.060277 [0 172.20.65.196:6379] "SET" "app|managed-resources|<application>|1.8.3.gz"
1697046400.521017 [0 10.50.80.110:50238] "get" "app|managed-resources|<application>|1.8.3"

Note the lack of the suffix on the last one.

I could potentially add a ton of logging, build argo-cd and create an image, and deploy that to get more information, but I was hoping to see if anyone has dealt with this or knows what might be happening first. Looking at the code, I can't see why the key formation would ever be inconsistent.

Some relevant code paths:

• The only place where I can see some potential inconsistency with compression settings: https://github.com/argoproj/argo-cd/blob/v2.8.4/cmd/argocd/commands/headless/headless.go#L247
• Key formation logic: https://github.com/argoproj/argo-cd/blob/v2.8.4/util/cache/redis.go#L54
• Redis Get call: https://github.com/argoproj/argo-cd/blob/v2.8.4/util/cache/redis.go#L118

[crenshaw-dev]

Are you passing any flags on the argocd app diff/manifests command, or just the app name?

[ashinsabu3]

I have seen the same when I upgraded from argo 2.7.2 to argo 2.8.2, setting the redis.compression config to none seems to fix this.

[cmontemuino]

Also confirmed that adding redis.compression: none in ConfigMap argocd-cmd-params-cm solves the issue. I'm also using ArgoCD v2.8.4.

[purajit]

Are you passing any flags on the argocd app diff/manifests command, or just the app name?

Hey apologies, really need to figure out my GH notifications game.

Not passing in any other flags to the argocd commands. I don't have a cluster without compression available right now, but I believe I had this issue with and without --local as well.

[tooptoop4]

same issue in 2.10.3. redis.compression: none as workaround solves