Deployment.apps "argocd-server" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: "var-files", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "plugins"]

[akumar-99]

Describe the bug Error: UPGRADE FAILED: cannot patch "argocd-server" with kind Deployment: Deployment.apps "argocd-server" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: "var-files", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "plugins"]

Where am I going wrong here?

To Reproduce Steps to reproduce the behavior:

1. Use helm chart - https://artifacthub.io/packages/helm/argo/argo-cd/5.39.0
2. values.yaml

global:
  # Default image used by all components
  image:
    # -- If defined, a repository applied to all Argo CD deployments
    repository: quay.io/argoproj/argocd
    # -- Overrides the global Argo CD image tag whose default is the chart appVersion
    tag: "latest"
    # -- If defined, a imagePullPolicy applied to all Argo CD deployments
    imagePullPolicy: IfNotPresent

  # Default logging options used by all components
  logging:
    # -- Set the global logging format. Either: `text` or `json`
    format: text
    # -- Set the global logging level. One of: `debug`, `info`, `warn` or `error`
    level: info

  # -- Annotations for the all deployed Statefulsets
  statefulsetAnnotations: {}

  # -- Annotations for the all deployed Deployments
  deploymentAnnotations: {}

  # -- Annotations for the all deployed pods
  podAnnotations: {}

  # -- Labels for the all deployed pods
  podLabels: {
    eks.amazonaws.com/fargate-profile: argocd
  }

  # -- Default tolerations for all components
  tolerations: 
  - key: "eks.amazonaws.com/compute-type"
    operator: "Exists"
    effect: "NoSchedule"

server:
  service:
    # -- Server service annotations
    annotations: {}
    # -- Server service labels
    labels: {}
    # -- Server service type
    type: NodePort
    # -- Server service http port for NodePort service type (only if `server.service.type` is set to "NodePort")
    nodePortHttp: 30080
    # -- Server service https port for NodePort service type (only if `server.service.type` is set to "NodePort")
    nodePortHttps: 30443
    # -- Server service http port
    servicePortHttp: 80
    # -- Server service https port
    servicePortHttps: 443
    # -- Server service http port name, can be used to route traffic via istio
    servicePortHttpName: http
    # -- Server service https port name, can be used to route traffic via istio
    servicePortHttpsName: https
    # -- Server service https port appProtocol. (should be upper case - i.e. HTTPS)
    # servicePortHttpsAppProtocol: HTTPS
    # -- LoadBalancer will get created with the IP specified in this field
    loadBalancerIP: ""
    # -- Source IP ranges to allow access to service from
    loadBalancerSourceRanges: []
    # -- Server service external IPs
    externalIPs: []
    # -- Denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints
    externalTrafficPolicy: ""
    # -- Used to maintain session affinity. Supports `ClientIP` and `None`
    sessionAffinity: ""
  
  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}

  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.16.1
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  # -- Additional containers to be added to the server pod
  ## Note: Supports use of custom Helm templates
  extraContainers:
    - name: avp
      command: [/var/run/argocd/argocd-cmp-server]
      image: registry.access.redhat.com/ubi8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp

        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: avp.yaml
          name: cmp-plugin

        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

configs:
  # ConfigMap for Config Management Plugins
  # Ref: https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/
  cmp:
    # -- Create the argocd-cmp-cm configmap
    create: true

    # -- Annotations to be added to argocd-cmp-cm configmap
    annotations: {}

    # -- Plugin yaml files to be added to argocd-cmp-cm
    plugins:
      argocd-vault-plugin:
        allowConcurrency: true
        generate:
          command:
            - argocd-vault-plugin
            - generate
            - "."
        discover:
          find:
            command:
              - sh
              - "-c"
              - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
        lockRepo: false

Expected behavior The upgrade should be completed successfully with the vault plugin set up.

Screenshots/Verbose output Error: UPGRADE FAILED: cannot patch "argocd-server" with kind Deployment: Deployment.apps "argocd-server" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: "var-files", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "plugins"]

Additional context Add any other context about the problem here.