nil pointer dereference after creating Custom Resource before CRD was synced

[sthomson-wyn]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Attemped to sync a custom resource in one Application, before the CRD was created by another Application. This resulted in repeated nil pointer dereferences. The issue was resolved by terminating the affected application controller, and letting it restart.

To Reproduce

I have not attempted a reproduction, but putting a custom resource in an application, then syncing the CRD in another application after syncing the initial app may result in this

Expected behavior

No attempt to dereference nil pointer!

Version

argocd: v2.8.4+b3ba6e1.dirty
  BuildDate: 2023-10-27T16:28:37Z
  GitCommit: b3ba6e1844548be490de550eb56fd3b4e2ea93a0
  GitTreeState: dirty
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:12:09Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.1.0 2023-06-19T16:58:18Z
  Helm Version: v3.12.1+gf32a527
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

Logs

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 257 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1337 +0x65
panic({0x3410220, 0x69c98e0})
	/usr/local/go/src/runtime/panic.go:890 +0x263
github.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0xc02dcfb640)
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:153 +0xe5
github.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0xc00940fc58?, 0xc02dcfb900?, 0x5?, {0x3a431a3?, 0xc007a60730?})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:133 +0x5a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc003f0aa18, 0xc003f0aa20, {0xc02dcfb900, 0x5, 0x5})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:101 +0x1c8
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc062edf000, 0xab, 0xc000bc4ab6?}, {0xc062ede800, 0xab?, 0x0?}, {0xc02dcfb900, 0x5, 0x5})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:646 +0x12b
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc05ad0ac00?, 0xc008236f80?, 0x3a66837?}, {0xc062ede000?, 0x326c980?, 0x0?}, {0x4c27718, 0xc007f3e7e0?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:271 +0x615
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000c262c0, 0xc052c0f000, 0xc02d8f6240, {0xc089e274d0, 0x1, 0x1}, {0xc01556ecb0?, 0x1, 0x1}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:581 +0x36d2
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0002ec900)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1444 +0x105e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:740 +0x29
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x6576222c22636e79?, {0x4be7780, 0xc001d12d20}, 0x1, 0xc000103980)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x67617373656d222c?, 0x3b9aca00, 0x0, 0x6e?, 0x6974696e692d7265?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x65696c7070612d65?, 0x656d616e222c2264?, 0x7473756c63223a22?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x25
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:739 +0x6be

[hamelg]

I get the same stacktrace with one of my argocd applications. Panic occurs when triggering a "Hard Refresh". Restarting the application controller does not fix the problem, the panic returns after restart. I only encounter this problem with a specific application that contains huge crd objects. The application controller panics only when this application is rendered with a cmp plugin. When not handled by a cmp plugin, the panic does not occur and "Hard refresh" works correctly.

Version

{
    "Version": "v2.8.4+ebab8ec",
    "BuildDate": "2023-10-30T19:24:59Z",
    "GitCommit": "ebab8ec6259a6997fa3f310cddc539cb0c76b442",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.1.0 unknown",
    "HelmVersion": "v3.12.1+g1b5d1f0",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0",
    "ExtraBuildInfo": "{Vendor Information: Red Hat OpenShift GitOps version: v1.10.1}"
}

Logs

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 236 [running]:
runtime/debug.Stack()
        /usr/lib/golang/src/runtime/debug/stack.go:24 +0x65
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
        /remote-source/argo_cd/app/controller/appcontroller.go:1337 +0x65
panic({0x341cb20, 0x6a0b2e0})
        /usr/lib/golang/src/runtime/panic.go:890 +0x263
github.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0xc0131f3640)
        /remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:153 +0xe5
github.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0xc0002e7ae0?, 0xc0131f3900?, 0x5?, {0x3a52a3b?, 0xc0135c2460?})
        /remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:133 +0x5a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc00c825d20, 0xc00c825d28, {0xc0131f3900, 0x5, 0x5})
        /remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:101 +0x1c8
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc006f19400, 0x41, 0xc000b22de4?}, {0xc006f19000, 0x41?, 0x0?}, {0xc0131f3900, 0x5, 0x5})
        /remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:646 +0x12b
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc009fb6d80?, 0xc000775900?, 0x3a7613f?}, {0xc006f18c00?, 0x3278840?, 0x7?}, {0x4c39318, 0xc0013a81b0?})
        /remote-source/argo_cd/app/util/argo/diff/diff.go:271 +0x615
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0007c2b00, 0xc001919000, 0xc00168f440, {0xc001920190, 0x1, 0x1}, {0xc0007a1ce0?, 0x1, 0x1}, 0x1, ...)
        /remote-source/argo_cd/app/controller/state.go:581 +0x36d2
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0002e8600)
        /remote-source/argo_cd/app/controller/appcontroller.go:1444 +0x105e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
        /remote-source/argo_cd/app/controller/appcontroller.go:740 +0x29
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x4bf9160, 0xc000ce0270}, 0x1, 0xc0006e9140)
        /remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
        /remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x25
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run
        /remote-source/argo_cd/app/controller/appcontroller.go:739 +0x6be

[hamelg]

My problematic argocd application crash the controller only when spec.syncPolicy.syncOptions has the flag ServerSideApply=true. Replacing this flag with Replace=true solves the issue.

[jgwest]

Likely a dupe of https://github.com/argoproj/argo-cd/issues/14098