argo-cd icon indicating copy to clipboard operation
argo-cd copied to clipboard

Published 20 hours ago • verified publisher icon argoproj

Panic with ServerSideDiff=true (but without IncludeMutationWebhook=true)

Open jackmaninov opened this issue 9 months ago • 13 comments

Checklist:

  • [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • [x] I've included steps to reproduce the bug.
  • [X] I've pasted the output of argocd version.

Describe the bug

While attempting to enable ServerSideDiff in order to get the green checkmark on my deployment of kube-prometheus-stack:57.1.1, I encountered a panic which I've pasted below.

After enabling ServerSideDiff=true via the web interface, my application gets stuck refreshing, and manually attempting to sync gives the panic notification, though my logs indicate it occurs almost immediately after enabling the annotation.

I noticed the reference to removeWebhookMutation in the stack trace, turned on IncludeMutationWebhook=true as well and everything worked as expected.

I was able to reproduce this on my install even on very simple applications. I'm sure more information about my config will be relevant, but I would need some guidance on what to contribute.

To Reproduce

  1. Add annotation to an application via the web interface: argocd.argoproj.io/compare-options: ServerSideDiff=true. I did this by going to Details -> Edit, then added a new line under annotations
  2. Press "Refresh".
  3. Press "Sync".

Expected behavior

Perhaps an error of some sort, but not a panic.

Screenshots

Version

argocd: v2.10.8+37b1cf5
  BuildDate: 2024-04-26T15:31:25Z
  GitCommit: 37b1cf5306f9c245f188c4c0566c23a0f80cdc65
  GitTreeState: clean
  GoVersion: go1.22.2
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.10.8+37b1cf5

Logs

Please excuse them being a bit munged, they came out of my logs as a single line.

time="2024-04-29T20:20:44Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 206 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
panic({0x3658d60?, 0x7189d60?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc001aa5f00, 0xc0097f4e18, 0x0, {0x50b4020?, 0x3d7cb48?})
	/go/pkg/mod/github.com/argoproj/[email protected]
0122213038-792124280fcc/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc0097f4de0, 0xc0097f4e18, {
0xc0054198a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd
	github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc0097f4e18?, 0xc0054198a8?, {0xc0054198a8?, 0x8?, 0xc011231ae0?})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc0097f4af0, 0xc0097f4af8, {0xc0054198a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc008425400, 0x29, 0x4197b3?}, {0xc008425200
, 0x29?, 0xe630240?}, {0xc0054198a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc00dca0840?, 0xc00f501d60?, 0x1e?}, {0xc008425000?, 0xc
00925b500?, 0x11?}, {0x50c90c0, 0xc00b974000?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00062a8c0, 0xc009c0c800, 0xc00652a240, {0xc00a04d140, 0x1, 0x1}, {0xc00925b4
90?, 0x1, 0x1}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/controller
.(*ApplicationController).processAppRefreshQueueItem(0xc000bac380)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1581 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util
/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x5071dc0, 0xc000c02330}, 0x1, 0xc000e4e660)
	/go/pkg/mod/
k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 127
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ce

jackmaninov avatar Apr 29 '24 20:04 jackmaninov

I've run into the exact same situation today

{
    "Version": "v2.10.8+37b1cf5",
    "BuildDate": "2024-04-26T13:20:04Z",
    "GitCommit": "37b1cf5306f9c245f188c4c0566c23a0f80cdc65",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.3+gf03cc04",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

Set http://argocd.argoproj.io/compare-options: ServerSideDiff=true annotation on an app and this results in the following

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 206 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()n	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
panic({0x3658d60?, 0x7189d60?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
githu
b.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc01c6acd70, 0xc01b7ae850, 0x0, {0x50b4020?, 0x3d7cb48?})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc01b7ae7f0, 0xc01b7ae85
0, {0xc006fc18a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc01b7ae850?, 0xc003a718a8?, {0xc003a718a8?, 0x8?, 0xc0075e0d70?})
	/go/pkg/mod/github.com/argoproj/gitops-engi
[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc01b7ae178, 0xc01b7ae180, {0xc003a718a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engin
e/pkg/diff.DiffArray({0xc00dbf8600, 0x29, 0x7fb710832a68?}, {0xc00dbf8400, 0x29?, 0xc01a3ce4e0?}, {0xc006fc18a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc011890580?, 0xc0
22c63460?, 0x1e?}, {0xc00dbf8200?, 0xc00c588380?, 0xe?}, {0x50c90c0, 0xc01b343680?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00075e8c0, 0xc00ca36800, 0xc012bf2240, {0xc00c2ca500, 0x5, 0x8}, {0xc019ec400
0?, 0x5, 0x8}, 0x1, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0007da700)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1581 +0x1223
github.com/argoproj/argo-cd/v2
/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?,
 {0x5071dc0, 0xc000dc6240}, 0x1, 0xc000082d20)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.
Until(0x0?, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 99
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ce

Add IncludeMutationWebhook=true to the annotation and the app successfully syncs

sherif-fanous avatar Apr 30 '24 17:04 sherif-fanous

@jackmaninov I just tested the kube-prometheus-stack helm chart applying the following application in my local kubernetes cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
spec:
  project: default
  source:
    chart: kube-prometheus-stack
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 51.2.0
    helm:
      valuesObject:
        kubeStateMetrics:
          enabled: false
        nodeExporter:
          enabled: false
        grafana:
          enabled: false
        alertmanager:
          enabled: false
        serviceMonitor:
          enabled: true
          prometheusSpec:
            serviceMonitorSelectorNilUsesHelmValues: false
  destination:
    namespace: monitoring
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

I wasn't able to reproduce the error. The application is synced as expected and no errors are returned in the UI or in the logs. I tested with Argo CD master build. Will run another test with 2.10.8 and confirm.

Can you please validate with the application above? It would be great if you could provide a minimal Application yaml that I can apply locally to try to reproduce.

leoluz avatar May 02 '24 21:05 leoluz

Set http://argocd.argoproj.io/compare-options: ServerSideDiff=true annotation on an app and this results in the following

@sherif-fanous not sure if this is just a typo in your message but the correct annotation is: argocd.argoproj.io/compare-options: ServerSideDiff=true and not http://argocd.argoproj.io/compare-options: ServerSideDiff=true

Please confirm.

leoluz avatar May 02 '24 21:05 leoluz

Can you please validate with the application above? It would be great if you could provide a minimal Application yaml that I can apply locally to try to reproduce.

Yes, that application causes the panic as well. The only modification I made was in argocd's namespace (not the destination namespace), which is instead something like namespace: prod-argocd (which I don't believe should matter, but you never know).

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 264 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(
*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
p
anic({0x3658d60?, 0x7189d60?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Ty
pe(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.remo
veWebhookMutation(0xc001d7b090, 0xc005680638, 0x0, {0x50b4020?, 0x3d7cb48?})
	/go/pkg/mod/github.com/argoproj/[email protected]
0122213038-792124280fcc/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc005680550, 0xc005680638, {
0xc0014d38a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd\
ngithub.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc005680638?, 0xc0014d38a8?, {0xc0014d38a8?, 0x8?, 0xc00e508eb0?})
	/go/pkg/m
od/github.com/argoproj/[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg
/diff.Diff(0xc00d405ec0, 0xc004f320e8, {0xc0014d38a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]
92124280fcc/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc000600c00, 0x41, 0x7ffa57cadf18?}, {0xc000
600800, 0x41?, 0xc00902cea0?}, {0xc0014d38a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]
fcc/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc00875b8c0?, 0xc01005a840?, 0x1e?}, {0xc019003
800?, 0xc00c862b60?, 0xf?}, {0x50c90c0, 0xc01134e9c0?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.co
m/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00062a8c0, 0xc00cfc3000, 0xc00a4b0000, {0xc0078076c0, 0x1, 0x1}, {0x
c00c862af0?, 0x1, 0x1}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/co
ntroller.(*ApplicationController).processAppRefreshQueueItem(0xc000bac380)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.
go:1581 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/cont
roller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/
pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x5071dc0, 0xc000ab4a80}, 0x1, 0xc000e4e660)
	/go/
pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 
0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?
, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*
ApplicationController).Run in goroutine 127
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ce

jackmaninov avatar May 02 '24 21:05 jackmaninov

Set http://argocd.argoproj.io/compare-options: ServerSideDiff=true annotation on an app and this results in the following

@sherif-fanous not sure if this is just a typo in your message but the correct annotation is: argocd.argoproj.io/compare-options: ServerSideDiff=true and not http://argocd.argoproj.io/compare-options: ServerSideDiff=true

Please confirm.

Sorry that was a typo. I used the correct annotation

sherif-fanous avatar May 03 '24 02:05 sherif-fanous

@jackmaninov where are you testing this? Is this running in a local Kubernetes cluster? Which Kubernetes version are you running?

leoluz avatar May 03 '24 14:05 leoluz

@jackmaninov where are you testing this? Is this running in a local Kubernetes cluster? Which Kubernetes version are you running?

This is in my (remote) development environment, running under Talos Linux v1.7.0, k8s version v1.30.0.

Might be a bit too bleeding edge, but since @sherif-fanous is seeing this as well maybe it's not so k8s version specific.

jackmaninov avatar May 03 '24 14:05 jackmaninov

After a bit more digging, I find that there's a failure to call a webhook repeatedly while the application is in the bad state (paniced and stuck refreshing), so the panic could be in reaction to that:

From my kube-apiserver log, this repeats rapidly:

E0503 14:48:24.681458 1 dispatcher.go:213] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://monitoring-kube-prome-operator.monitoring.svc:443/admission-prometheusrules/validate?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

I'll try to track down what's causing the bad certificate and see if it improves things.

jackmaninov avatar May 03 '24 14:05 jackmaninov

@sherif-fanous can you please confirm in which environment you are running your tests and what is the Kubernetes server version?

leoluz avatar May 03 '24 20:05 leoluz

@leoluz I'm using Talos Linux 1.7 running Kubernetes 1.30.0. This is a home lab cluster.

sherif-fanous avatar May 03 '24 22:05 sherif-fanous

Ok, small world...

jackmaninov avatar May 04 '24 03:05 jackmaninov

And, yes, I'm (now) aware argo-cd is only tested up to k8s v1.28 as per: https://argo-cd.readthedocs.io/en/stable/operator-manual/tested-kubernetes-versions/

jackmaninov avatar May 04 '24 05:05 jackmaninov

I was having a TLS error (invalid cipher suite specified) during deployment and I had to apply this patch (https://github.com/argoproj/argo-cd/pull/17569) on top of Argo CD 2.10.9 to make it work.

I tested the patched version against Kubernetes 1.30 and wasn't able to reproduce the issue. I applied the prometheus-stack helm and it is working as expected.

@jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from master branch and let me know if it addresses the issue?

leoluz avatar May 06 '24 21:05 leoluz

I got a similar panic with 2.10.7. I downgraded to 2.9.12, and did another sync and the issue went away. Maddeningly enough, after upgrading back to 2.10.7, deleting the application and trying again, I could not reproduce this issue.

github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1466 +0x50\npanic({0x2f89740?, 0x6a4bdc0?})\n
\t/usr/local/go/src/runtime/panic.go:920 +0x26c\ngithub.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0x4000bb54c0)\n\t/go/pkg/mod/github.com/argoproj/[email protected]/pk
g/diff/diff.go:321 +0x9c\ngithub.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0x40015afe90?, 0x4000bb5858?, 0x8?, {0x36ad003?, 0x49e6390?})\n\t/go/pkg/mod/github.com/argoproj/[email protected]
122213038-792124280fcc/pkg/diff/diff.go:301 +0x4c\ngithub.com/argoproj/gitops-engine/pkg/diff.Diff(0x40015af670, 0x40015af678, {0x4000bb5858, 0x8, 0x8})\n\t/go/pkg/mod/github.com/argoproj/[email protected]
40122213038-792124280fcc/pkg/diff/diff.go:110 +0x234\ngithub.com/argoproj/gitops-engine/pkg/diff.DiffArray({0x400827dc00, 0x7e, 0x4000bb57d8?}, {0x400827d800, 0x7e?, 0xa?}, {0x4000bb5858, 0x8, 0x8})\n\t/go/pkg/mod/
github.com/argoproj/[email protected]/pkg/diff/diff.go:814 +0x11c\ngithub.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0x400b90e800?, 0x40011992e0?, 0x36ffd83?}, {0x400827c800
?, 0x4000f76b60?, 0x2?}, {0x49fa270, 0x4001ef7ad0?})\n\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:310 +0x5a8\ngithub.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0x4000f777
a0, 0x400170ec00, 0x4007e40240, {0x4007b7e5c0, 0x2, 0x2}, {0x4000f76b60?, 0x2, 0x2}, 0x0?, ...)\n\t/go/src/github.com/argoproj/argo-cd/controller/state.go:649 +0x2e24\ngithub.com/argoproj/argo-cd/v2/controller.(*Ap
plicationController).processAppRefreshQueueItem(0x4000a74700)\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1577 +0xe44\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.f
unc3()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:815 +0x2c\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:157
+0x40\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x49a3f00, 0x400162c900}, 0x1, 0x40009a5980)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0x90\nk8s.io/apimachinery/pkg/util/wa
it.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x80\nk8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)\n\t/go/pkg/mod/k8s.io/apimach
[email protected]/pkg/util/wait/wait.go:92 +0x28\ncreated by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 114\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:8
14 +0x5f0\n","time":"2024-05-08T11:44:38Z"}

The appset used:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: kube-prometheus-stack
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - name: blake
            chartRevision: 55.1.0
            valuesRevision: main
  template:
    metadata:
      name: '{{.name}}-kube-prometheus-stack'
    spec:
      project: default
      sources:
        - chart: kube-prometheus-stack
          repoURL: https://prometheus-community.github.io/helm-charts
          targetRevision: '{{.chartRevision}}'
          helm:
            releaseName: kube-prometheus-stack
            valueFiles:
              - $values/values.yaml
        - repoURL: https://github.com/blakepettersson/argocd-kube-prometheus-stack
          targetRevision: '{{.valuesRevision}}'
          ref: values
      destination:
        namespace: monitoring
        name: '{{.name}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true

kubectl (I know I need to bump my k3s, it's way too old 😄)

kubectl version
Client Version: v1.29.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.23.5+k3s1
WARNING: version difference between client (1.29) and server (1.23) exceeds the supported minor version skew of +/-1

Working Argo CD version:

argocd version
argocd-server: v2.9.12+487dce5.dirty
  BuildDate: 2024-04-19T03:54:04Z
  GitCommit: 487dce58bfa228a4f8bc22c626d1920f1617c066
  GitTreeState: dirty
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.24.17

Non-working Argo CD version:

argocd-server: v2.10.7+96c0c41.dirty
  BuildDate: 2024-04-19T03:53:43Z
  GitCommit: 96c0c41b8948b0738567bd5c748a3de0cec5d19e
  GitTreeState: dirty
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

blakepettersson avatar May 08 '24 12:05 blakepettersson

@jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from master branch and let me know if it addresses the issue?

How do we do that? Set image tag to latest?

sherif-fanous avatar May 08 '24 14:05 sherif-fanous

@jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from master branch and let me know if it addresses the issue?

I managed this today and running master did not resolve the issue.

I did however try deploying the test application in a freshly set-up local Talos cluster, and the issue did not present itself using argo-cd v2.11.0, so it is something more complicated going on with my setup. I suspect the failing admission webhook error I noted above, which I have still not tracked down.

EDIT: @sherif-fanous: deploying master involves pulling the git repo, running docker build, uploading the image to a repo somewhere, setting global.image.registry and global.image.tag in the helm values (in my case I installed with helm), then patching in an imagePullSecret into each of argo's ServiceAccounts.

jackmaninov avatar May 08 '24 21:05 jackmaninov

@blakepettersson Can you please take a look at https://github.com/argoproj/gitops-engine/pull/574 It addresses the panic issue. However the reason why the ParseableType can not resolve the GVK is still unknown.

leoluz avatar May 09 '24 14:05 leoluz

@leoluz Can this issue please be re-opened?

I built master and still getting the panic

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 236 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1485 +0x54
panic({0x37ee8c0?, 0x76c5910?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc00c5b6530, 0xc00c5b6250, 0x0, {0x552c700?, 0x3f1dff0?})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc00c5b61e0, 0xc00c5b6250, {0xc00dc5f898, 0x8, 0x8})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc00c5b6250?, 0xc00dc5f898?, {0xc00dc5f898?, 0x8?, 0xc00de232c0?})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc00cece000, 0xc00cece008, {0xc00dc5f898, 0x8, 0x8})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:88 +0x1cf
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc00bdcc800, 0x33, 0x7?}, {0xc00bdcc600, 0x33?, 0xc00a629f80?}, {0xc00dc5f898, 0x8, 0x8})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:821 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc004ab71e0?, 0x3c5ae00?, 0x1e?}, {0xc00f386800?, 0xc00c6fe7e0?, 0x7?}, {0x5542f20, 0xc00932f680})
        /go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74e
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0005f8e00, 0xc00737ec08, 0xc00a205448, {0xc00dc1db00, 0x4, 0x4}, {0xc00dc54e00, 0x4, 0x4}, 0x0, ...)
        /go/src/github.com/argoproj/argo-cd/controller/state.go:684 +0x3d3f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000789880)
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1595 +0x1183
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:833 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000de4df0, {0x54e88e0, 0xc00119f1d0}, 0x1, 0xc000dd8de0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000de4df0, 0x3b9aca00, 0x0, 0x1, 0xc000dd8de0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 125
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:832 +0x865

I've not used Go in a long time but looking at the trace my understanding is that the panic is resulting from /go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43 which is the first statement in the following method.

func (p *GvkParser) Type(gvk schema.GroupVersionKind) *typed.ParseableType {
	typeName, ok := p.gvks[gvk]
	if !ok {
		return nil
	}
	t := p.parser.Type(typeName)
	return &t
}

This would indicate to me that either p or p.gvks are nil

sherif-fanous avatar May 16 '24 18:05 sherif-fanous

Also having this issue.

Regarding config and versions shared in this thread, most things are the same but I'm using k3s 1.30.

Running the newest 2.9.15 version works but anything above that (2.10+) gives me this panic.

Nabsku avatar May 23 '24 12:05 Nabsku