argo-cd
argo-cd copied to clipboard
Inconsistency in redis key formation in ArgoCD 2.8.4 with compression enabled
Checklist:
- [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- [ ] I've included steps to reproduce the bug.
- [x] I've pasted the output of
argocd version
.
Describe the bug
ArgoCD 2.8 enables Redis compression by default. However, after upgrading, I found I was unable to run commands like argocd app diff/manifest
, since it would fail with
ERRO[0031] finished unary call with code Unknown error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-10-11T10:30:18-07:00" grpc.time_ms=1554.185 span.kind=server system=grpc
Everything else runs without issue - syncing, UI browsing, etc.
To reproduce Unsure what to mention here.
What I've tried that didn't help:
- verbose logging; but unfortunately nothing relevant shows up
- ensuring
REDIS_COMPRESSION
isgzip
on all pods - explicitly setting the
redis.compression
config togzip
- explicitly setting
--redis-compress=gzip
as a server argument
What works:
- explicitly setting the
redis.compression
config tonone
Current configuration:
- We use
--insecure --enable-gzip
(we handle TLS termination ourselves) - We use the
public.ecr.aws/docker/library/haproxy,redis
repositories, with HA enabled.
Expected behavior
Everything should "just work", and key formation should be consistent.
Version
We're using the ArgoCD helm chart (v5.46.7).
# on server pod
> /usr/local/bin/argocd-server version
argocd-server: v2.8.4+c279299
BuildDate: 2023-09-13T19:12:09Z
GitCommit: c27929928104dc37b937764baf65f38b78930e59
GitTreeState: clean
GoVersion: go1.20.6
Compiler: gc
Platform: linux/amd64
# on local
> argocd version
argocd: v2.8.4+c279299
BuildDate: 2023-09-13T19:43:37Z
GitCommit: c27929928104dc37b937764baf65f38b78930e59
GitTreeState: clean
GoVersion: go1.20.7
Compiler: gc
Platform: darwin/arm64
argocd-server: v2.8.4+c279299
BuildDate: 2023-09-13T19:43:37Z
GitCommit: c27929928104dc37b937764baf65f38b78930e59
GitTreeState: clean
GoVersion: go1.20.7
Compiler: gc
Platform: darwin/arm64
Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
Helm Version: v3.9.0+g7ceeda6
Kubectl Version: v0.24.2
Jsonnet Version: v0.20.0
Logs
I cleared out some cache keys, and then ran redis-cli monitor | grep <application>
on the Redis pods:
1697046390.058300 [0 172.20.65.196:6379] "SET" "app|managed-resources|<application>|1.8.3.gz"
1697046390.060277 [0 172.20.65.196:6379] "SET" "app|managed-resources|<application>|1.8.3.gz"
1697046400.521017 [0 10.50.80.110:50238] "get" "app|managed-resources|<application>|1.8.3"
Note the lack of the suffix on the last one.
I could potentially add a ton of logging, build argo-cd and create an image, and deploy that to get more information, but I was hoping to see if anyone has dealt with this or knows what might be happening first. Looking at the code, I can't see why the key formation would ever be inconsistent.
Some relevant code paths:
- The only place where I can see some potential inconsistency with compression settings: https://github.com/argoproj/argo-cd/blob/v2.8.4/cmd/argocd/commands/headless/headless.go#L247
- Key formation logic: https://github.com/argoproj/argo-cd/blob/v2.8.4/util/cache/redis.go#L54
- Redis Get call: https://github.com/argoproj/argo-cd/blob/v2.8.4/util/cache/redis.go#L118
Are you passing any flags on the argocd app diff/manifests
command, or just the app name?
I have seen the same when I upgraded from argo 2.7.2 to argo 2.8.2, setting the redis.compression
config to none seems to fix this.
Also confirmed that adding redis.compression: none
in ConfigMap argocd-cmd-params-cm
solves the issue. I'm also using ArgoCD v2.8.4.
Are you passing any flags on the
argocd app diff/manifests
command, or just the app name?
Hey apologies, really need to figure out my GH notifications game.
Not passing in any other flags to the argocd
commands. I don't have a cluster without compression available right now, but I believe I had this issue with and without --local
as well.
same issue in 2.10.3. redis.compression: none as workaround solves