argocd local diff fails `with invalid memory address or nil pointer dereference`

[arthurk]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Argocd diff with --local doesn't work as expected when using --grpc-web. It only works when port-forwarding and sending the request to the argocd-server directly (with --plaintext).

I have argo cd installed in two environments (staging and prod) but see a different error when running the diff.

In staging the error is:

$ argocd --server deploy.aaa.dev --grpc-web --auth-token xxx app diff datadog --server-side-generate --local staging
FATA[0002] rpc error: code = Unknown desc = error receiving manifest file stream: error receiving tgz file: file checksum validation error: calc e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sent 54913ce3761a279f442b7c4a248b135eb86ab9594fdbd50c2b4e9b9437aabc3e

When port-forwarding to the argo cd server and using --plaintext it works fine:

argocd --server localhost:8080 --plaintext --auth-token xxx app diff datadog --server-side-generate --local staging

===== apps/DaemonSet datadog/datadog ======
1349c1349
<           value: 111
---
>           value: 222
...

In production environment running with --grpc-web on the domain I see a different error:

argocd --server deploy.aaa.com --grpc-web --auth-token xxx app diff datadog --server-side-generate --local prod
FATA[0002] failed to send manifest stream file: error sending stream: EOF

In the server logs it shows the panic error:

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 201 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1312 +0x65
panic({0x2f3b200, 0x5f7d1b0})
	/usr/local/go/src/runtime/panic.go:844 +0x258
github.com/argoproj/argo-cd/v2/util/argo/managedfields.newTypedResults(0xc00c1fb490?, 0xc00c1fb4a8, 0x0)
	/go/src/github.com/argoproj/argo-cd/util/argo/managedfields/managed_fields.go:94 +0x36
github.com/argoproj/argo-cd/v2/util/argo/managedfields.Normalize(0xc00c1fb498, 0xc00c1fb490, {0xc007bdcfb0, 0x1, 0xc00d4c1cd0?}, 0x7?)
	/go/src/github.com/argoproj/argo-cd/util/argo/managedfields/managed_fields.go:31 +0xb4
github.com/argoproj/argo-cd/v2/util/argo/diff.preDiffNormalize({0xc012a1ee60, 0x4, 0x1d7e693?}, {0xc012a1ee40, 0x4, 0x1b?}, {0x4523f40, 0xc00d1f62d0})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:362 +0x57b
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc012a1ee60?, 0xc000059160?, 0x34f01e6?}, {0xc012a1ee40?, 0xc000b4d420?, 0x7?}, {0x4523f40, 0xc00d1f62d0?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:241 +0x56
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000010420, 0xc008b8cc00, 0xc010632900, {0xc000b4d450, 0x4}, {{0xc001007ec0, 0x35}, {0xc001007e80, 0x32}, {0xc000b4d450, ...}, ...}, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:495 +0x2b1a
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000dffa00)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1398 +0x78b
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:726 +0x29
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x44eb500, 0xc000d62870}, 0x1, 0xc00005f7a0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x25
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:725 +0x6ba

When using port-forward to the argocd-server with --plaintext it also works fine.

Both cluster are running the same argo cd version.

To Reproduce

Run

$ argocd --server argocd.mycompany.com --grpc-web --auth-token xxx app diff backend-app --server-side-generate --local main-repo

Expected behavior

See a diff, or at least a clear error message that shows what's wrong.

Screenshots

Version

argocd: v2.5.6
  BuildDate: unknown
  GitCommit: v2.5.6
  GitTreeState: clean
  GitTag: v2.5.6
  GoVersion: go1.19.4
  Compiler: gc
  Platform: linux/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
argocd-server: v2.5.5+fc3eaec

[fabricepipart1a]

I think we are suffering from the same issue. Each time I use the server-side-generate option I get :

> argocd app diff myapp --local ./myapp/chart --server-side-generate                      
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
FATA[0003] rpc error: code = Unknown desc = error receiving manifest file stream: error receiving tgz file: file checksum validation error: calc e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sent 142883fa923efe9d2cbb409eb2e540c65b3d93b28a68c3afdcdd4b0d8181444c 

As a reference:

argocd: v2.5.7+e0ee345.dirty
  BuildDate: 2023-01-18T04:25:01Z
  GitCommit: e0ee3458d0921ad636c5977d96873d18590ecf1a
  GitTreeState: dirty
  GoVersion: go1.19.5
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.5.6+9db2c94
  BuildDate: 2023-01-10T19:30:17Z
  GitCommit: 9db2c9471f6ff599c3f630b446e940d3a065620b
  GitTreeState: clean
  GoVersion: go1.18.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

This is a bit scary if this becomes the default behavior in 2.6.

[batazor]

I observe the same behavior in 2.7.0-rc1

my case: I updated the prometheus-operator and recreated a couple of crd's. I got the error described above. Then I re-created the crd again and updated the synchronization - this fixed the error

[fabricepipart1a]

Dear maintainers, is there anything I can provide to help you troubleshoot the error I get?

[jccarte]

I am pretty sure I am seeing the same thing. It works if I log in via --core (direct kubernetes access) but fails w/ the same error error receiving manifest file stream: error receiving tgz file: file checksum validation **error** When going through cloudflare

It appears that it is creating an empty file on the argocd-repo-server during these failures as well.

[ggirard07]

Same error than @fabricepipart1a here when mixing argocd app diff --server-side-generate while argocd login used --grpc-web!

[maxbrunet]

I think the reason is client-side streaming is not supported by GRPC-Web, only unary calls: improbable-eng/grpc-web#client-side-streaming

https://github.com/argoproj/argo-cd/blob/e233334a4ba43eb74f5380a3197753c7afffdb86/server/application/application.proto#L337-L343

This would require a different approach for this use case, like sending the chunks via multiple unary calls and reassembling them server-side.

[crenshaw-dev]

@maxbrunet good find. Should we just have the CLI error out when both --server-side-generate and --grpc-web are specified?

[maxbrunet]

A clear error message would surely improve the user experience. github.com/improbable-eng/grpc-web also supports upgrading Websocket to gRPC streaming, maybe it could be the alternative for this case.

Note that @improbable-eng/grpc-web provides a built-in websocket transport that can support client-side/bi-directional streaming RPCs.

[tspadi]

Hello there, I'm working on a project using argocd api (v2.9.9), in a very similar way than argocd cli. I'm facing the same issue when sending local files to argocd server to generate manifests. Using grpc-web enabled or disabled client does not make any difference, I consistently have the following error message at each attempt : error receiving manifest file stream: error receiving tgz file: file checksum validation error: calc e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sent 656a1cc968ea5bb9ab6f15df888c7d84553462a900220b0df531d1d9c1d7d462" Meaning the server computes the checksum on an empty file, in other words, the sent archive file didn't make it to the server. I use SendApplicationManifestQueryWithFiles function to achieve this, in a similar way than here : As I am only dependent on server-side implementation, but have the possibility to use a different way to implement this part on client side, would it be possible to overcome this issue using other functions or even github.com/improbable-eng/grpc-web ? or this wouldn't be possible without fixing something server-side ?

[jeremych1000]

Can I find out what the latest is on this?

From what I'm gathering (I'm using traefik as ingress):

• I need to configure traefik ingressroute instead of using ingress - done
• I need to not use --grpc-web with --local - done
• I need to use --server-side-generate - done
• I need to use --local-include "*.*" to include raw files outside my templates - done
• I need to use the root directory in --local and not the chart directory - done

I'm still getting both these errors and haven't been able to figure out a combination that works for server side diffs that includes Kubernetes schema validation!

FATA[0008] failed to send manifest stream file: error sending stream: EOF

or

FATA[0004] rpc error: code = Unknown desc = error receiving manifest file stream: error receiving tgz file: file checksum validation error