ArgoCD upgrade from Helm Chart 6.7.3 to 7.3.11 fails due to argo-redis secret job

[bhavinkotak]

Describe the bug

We have deployed ArgoCD in AKS Cluster 1.27. Redis-HA mode is enabled from argo-cd helm chart. Argo is getting self managed. So installation/upgrade is done by updating ArgoCD app-set.yaml and not via helm upgrade command.

We tried upgrading ArgoCD using helm chart from v6.7.3 to v7.3.11. It tries to upgrade itself; however, there is a job (Name: argocd-redis-secret-init; Kind: Job) which just hangs.

argocd-redis-secret-init-xxxxx POD logs -

Checking for initial Redis password in secret argocd/argocd-redis at key auth.
Argo CD Redis secret state confirmed: secret name argocd-redis.
Password secret is configured properly.

We are able to see new secret getting created. However, no other error message or information gets posted. We did restart all the deployment and sts resources for argocd; however no changes are observed.

Related helm chart

argo-cd

Helm chart version

7.3.11

To Reproduce

1. Install ArgoCD helm chart v6.7.3 - everything is up and running
2. Upgrade ArgoCD to helm chart v7.3.11 - it tries to upgrade itself; however there is a job (Name: argocd-redis-secret-init; Kind: Job) which just hangs.

argocd-redis-secret-init-xxxxx POD logs -

Checking for initial Redis password in secret argocd/argocd-redis at key auth. Argo CD Redis secret state confirmed: secret name argocd-redis. Password secret is configured properly.

We are able to see new secret getting created. However, no other error message or information gets posted. We did restart all the deployment and sts resources for argocd; however no changes are observed.

Expected behavior

Successful upgrade of ArgoCD using latest helm chart 7.3.11

Screenshots

No response

Additional context

No response

[AssenDimitrov]

Same, you got any workaround? I am using argocd in GKE, without network policies...

[bhavinkotak]

nah.. I am still stuck on this issue...

[jkleinlercher]

Does the job hang in PreSync hook in ArgoCD UI? I have the same problem posted in https://github.com/argoproj/argo-cd/issues/6880#issuecomment-2263802072

[jkleinlercher]

I think a ttl for the redis job could help

[yu-croco]

Maybe fixed in https://github.com/argoproj/argo-helm/pull/2861 ? Please feel free to reopen if situation is not fixed yet.

[speedythesnail]

This issue should be re-opened. I am unable to do a fresh install of ArgoCD using the helm chart. I am able to do so using the operator in OpenShift, but in my Kind cluster it fails using the chart.

[keithdwilliams]

Facing the same issue, this needs to be re-opened.

[lboclboc]

Facing the same issue with the helm chart version 7.5.2. The job for creating the argocd-redis secret is not run at so several containers is complaining about the missing secret.

[DoumLaberge]

I have a similar issue when I try ton install the Argocd Helm Chart version 7.5.1. I've try the 6.11.1 version and the problem is al;ready there. I've the redis HA mode. I have this error:

Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade argo-cd/templates/redis-secret-init/job.yaml failed: Job in version "v1" cannot be handled as a Job: json: cannot unmarshal string into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type []v1.LocalObjectReference

when I run it in debug mode, I can see there somthing not right in the argo-cd/templates/redis-secret-init/job.yaml There are a white line between the key iomagePullSecrets and the value

# Source: argo-cd/templates/redis-secret-init/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: argocd-redis-secret-init
  namespace: "argocd"
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation
  labels:
    helm.sh/chart: argo-cd-7.5.2
    app.kubernetes.io/name: argocd-redis-secret-init
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/component: redis-secret-init
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: "v2.12.3"
spec:
  ttlSecondsAfterFinished: 60
  template:
    metadata:
      labels:
        helm.sh/chart: argo-cd-7.5.2
        app.kubernetes.io/name: argocd-redis-secret-init
        app.kubernetes.io/instance: argocd
        app.kubernetes.io/component: redis-secret-init
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: argocd
        app.kubernetes.io/version: "v2.12.3"
    spec:
      imagePullSecrets:
        
        nexussecret
      containers:
      - command:
          - argocd
          - admin
          - redis-initial-password
        image: quay.io/argoproj/argocd:v2.12.3
        imagePullPolicy: IfNotPresent
        name: secret-init
        resources:
          {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
      restartPolicy: OnFailure
      serviceAccountName: argocd-redis-secret-init

[DoumLaberge]

The template redis-secret-init/job.yaml was introduce in 6.10.0 and doesn't seem to work since

[DoumLaberge]

I just find the error

{{- with .Values.global.imagePullSecrets }} imagePullSecrets: {{ toYaml . | nindent 8 }} {{- end }}

There a missing '-' in front of the toYaml in the redis-secret-init/job.yaml

[eltalkarim]

This is still happening

~ helm upgrade argocd argo/argo-cd --version 7.6.5 --wait -n argocd --debug 
upgrade.go:142: [debug] preparing upgrade for argocd
upgrade.go:524: [debug] copying values from argocd (v14) to new release.
upgrade.go:150: [debug] performing update for argocd
upgrade.go:322: [debug] creating upgraded release for argocd
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" ServiceAccount
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" Role
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" RoleBinding
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job argocd-redis-secret-init with timeout of 5m0s
client.go:568: [debug] Add/Modify event for argocd-redis-secret-init: ADDED
client.go:607: [debug] argocd-redis-secret-init: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
upgrade.go:434: [debug] warning: Upgrade "argocd" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:84: [debug] pre-upgrade hooks failed: timed out waiting for the condition

Logs from the init job:

│ argocd-redis-secret-init-jc9jp Checking for initial Redis password in secret argocd/argocd-redis at key auth.                                                                                                                                                                │
│ argocd-redis-secret-init-jc9jp Argo CD Redis secret state confirmed: secret name argocd-redis.                                                                                                                                                                               │
│ argocd-redis-secret-init-jc9jp Password secret is configured properly.   

using the default values here.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bhavinkotak]

issue still exists and seems to be discussed here - https://github.com/argoproj/argo-cd/issues/19798

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[lbm]

issue still exists and seems to be discussed here - argoproj/argo-cd#19798

This is still a problem and I can replicate it with a fresh install of the 7.8.2 Helm chart. This issue should be kept open.

[adalinesimonian]

~~Also running into this, I don't think this issue should be closed. Using 7.8.2 chart as well.~~

I was able to resolve my issue. It was PEBKAC. My issue is that the initial installation of ArgoCD was not done with the Helm chart, but through Kustomize. After nuking the Argo install and installing it with an older version but using the Helm chart, I tested upgrading to the latest chart version 7.8.2, and it worked. So at least in my case, that seems to have fixed the issue.