Secret key can not be omitted

[dtaniwaki]

Describe the bug

Although the comment in secret.yaml indicates server.secretkey can be auto-generated if it's missing. https://github.com/argoproj/argo-cd/blob/d3c850b8e7e67d1aa4c2deb6b77a4edbf4b7f261/docs/operator-manual/argocd-secret.yaml#L19

However, the code doesn't allow empty server.secretkey.

https://github.com/argoproj/argo-cd/blob/d3c850b8e7e67d1aa4c2deb6b77a4edbf4b7f261/util/settings/settings.go#L499

Which is the expected behavior?

To Reproduce

Just after installing ArgoCD and set admin.password (although this is not mentioned in the docs).

Expected behavior

Asking the expected behavior.

Screenshots

N/A

Version

$ argocd version
argocd: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:36Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:03Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

time="2019-07-14T11:45:39Z" level=info msg="Starting configmap/secret informers"
time="2019-07-14T11:45:39Z" level=info msg="Configmap/secret informer synced"
time="2019-07-14T11:45:39Z" level=fatal msg="server.secretkey is missing"

Have you thought about contributing a fix yourself?

I'm not sure which is the expected behavior.

[jessesuen]

The API server is supposed to generate this. During new installations, we will sometimes see application-controller crash because it started before the api-server had a chance to generate it.

Which service are your logs coming from?

[dtaniwaki]

The log came from the application controller.

[dtaniwaki]

Uh, the api server was not running because of PSP. I'll try it and update you again.

[dtaniwaki]

I confirmed it is created automatically if the api server is running.

[Ankitadev]

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

[alkdese]

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

In my situation I started having CrashLoopBackoff on argocd-dex-server in the logs I have "server.secretkey is missing" after deleting argocd namespace and creating it from scratch. I just now deleted already 3 times and recreated and still same problem. Any advice?

[hyerim-gn]

I solved the problem with node scale out due to the problem caused by the API server not being executed.

[emirot]

Seeing this issue after doing:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.7/manifests/install.yaml

Then CrashLoopBackOff on argocd-dex-server

kubectl logs argocd-dex-server-56d495844c-l2lxc -n argocd
time="2022-07-20T22:45:06Z" level=info msg="Starting configmap/secret informers"
time="2022-07-20T22:45:06Z" level=info msg="Configmap/secret informer synced"
time="2022-07-20T22:45:06Z" level=fatal msg="server.secretkey is missing"

[crenshaw-dev]

@emirot can you confirm that the API server is successfully coming up?

[emirot]

@crenshaw-dev sorry about the noise, I figured it out, it's not related, it is due to Open Policy Agent

[brianpooe]

@emirot i have the exact same issue how did you resolve yours?

[emirot]

@brianpooe In my case I had to do add to add --set global.networkPolicy.create=true in my helm install

[winston0410]

I have the same issue when I install argocd with Core Install. @dtaniwaki Can we reopen this issue?

[yevhen-harmonizehr]

i do have same issue after my redis pods restarted. My argo redis do not have PVC, so this might be related.

[TeamDman]

#10831 might be related

I had this

$ kubectl get all -n argocd
NAME                                                 READY   STATUS             RESTARTS      AGE
pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx   1/1     Running            0             17m
pod/argocd-argo-cd-repo-server-9565b9f86-5956t       1/1     Running            0             17m
pod/argocd-argo-cd-server-685dfc4dd7-kgl8r           0/1     CrashLoopBackOff   7 (33s ago)   11m
pod/argocd-redis-master-0                            1/1     Running            0             12m

NAME                                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/argocd-argo-cd-app-controller   ClusterIP   10.0.189.90   <none>        8082/TCP         17m
service/argocd-argo-cd-repo-server      ClusterIP   10.0.80.110   <none>        8081/TCP         17m
service/argocd-argo-cd-server           ClusterIP   10.0.42.249   <none>        80/TCP,443/TCP   17m
service/argocd-redis-headless           ClusterIP   None          <none>        6379/TCP         17m
service/argocd-redis-master             ClusterIP   10.0.51.188   <none>        6379/TCP         17m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-argo-cd-app-controller   1/1     1            1           17m
deployment.apps/argocd-argo-cd-repo-server      1/1     1            1           17m
deployment.apps/argocd-argo-cd-server           0/1     1            0           17m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-argo-cd-app-controller-6fbb4ccf8d   1         1         1       17m
replicaset.apps/argocd-argo-cd-repo-server-9565b9f86       1         1         1       17m
replicaset.apps/argocd-argo-cd-server-685dfc4dd7           1         1         0       17m

NAME                                   READY   AGE
statefulset.apps/argocd-redis-master   1/1     17m

$ kubectl logs -n argocd pod/argocd-argo-cd-server-685dfc4dd7-kgl8r
Defaulted container "argocd-server" out of: argocd-server, wait-for-redis (init)
 19:37:13.78 
 19:37:13.79 Welcome to the Bitnami argo-cd container
 19:37:13.79 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:37:13.79 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:37:13.79 
 19:37:13.80 INFO  ==> Configuring libnss_wrapper

time="2022-11-01T19:37:13Z" level=info msg="ArgoCD API Server is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd port=8080 version=v99.99.99+unknown
time="2022-11-01T19:37:13Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="Initialized server signature"
time="2022-11-01T19:37:14Z" level=info msg="admin disabled"
time="2022-11-01T19:37:14Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="configmap informer cancelled"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="secrets informer cancelled"
panic: server.secretkey is missing

goroutine 1 [running]:
github.com/argoproj/argo-cd/v2/util/session.NewSessionManager(0xc00004d980, {0x44c4aa0, 0xc000fcc580}, {0x34aec23, 0x16}, 0xc000aee5e0?, {0x44ddf10, 0xc0005a1f80})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/util/session/sessionmanager.go:124 +0x3fc
github.com/argoproj/argo-cd/v2/server.NewServer({_, _}, {0x0, 0x0, 0x1, {0x7ffc91ef54aa, 0x18}, 0x1f90, 0x1f93, {0xc000aee5e0, ...}, ...})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/server/server.go:267 +0x5af
github.com/argoproj/argo-cd/v2/cmd/argocd-server/commands.NewCommand.func1(0xc000afea00?, {0x347c98d?, 0xb?, 0xb?})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/argocd-server/commands/argocd_server.go:192 +0xfa8
github.com/spf13/cobra.(*Command).execute(0xc000afea00, {0xc0000ba010, 0xb, 0xb})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/[email protected]/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0xc000afea00)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/[email protected]/command.go:918
main.main()
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/main.go:57 +0x2c5

$ kubectl logs -n argocd pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx
Defaulted container "controller" out of: controller, wait-for-redis (init)
 19:21:17.44 
Welcome to the Bitnami argo-cd container
 19:21:17.44 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:21:17.45 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:21:17.45 
INFO  ==> Configuring libnss_wrapper
level=info msg="ArgoCD Application Controller is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd version=v99.99.99+unknown
level=info msg="Processing all cluster shards"
level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
level=info msg="Starting configmap/secret informers"
level=info msg="Configmap/secret informer synced"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=info msg="0xc000ff02a0 subscribed to settings updates"
level=info msg="Starting secretInformer forcluster"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Failed to save clusters info: server.secretkey is missing"
level=info msg="Notifying 1 settings subscribers: [0xc000ff02a0]"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"

Workaround

Manually specifying server.secretKey as an empty string in argocd-secret seems to have fixed the problem without having to manually restart anything after helm applying... idk if everything is actually working, but the pod stopped crashing and logs look healthy now

[gaeljw]

For anyone coming here for similar issue where server.secretKey is not defined, here's what I did to make it work back:

• delete the secret
• re-create the secret but empty
• restart argocd-server

[ali-de7]

I fixed it by simply restarting the server. Here's what I did to fix it:

kubectl rollout restart deploy/argocd-server

[crenshaw-dev]

Reopening because I think this is almost definitely a persistent problem with core install.

[pgpx]

Could we split argocd-secret into 2 or more secrets? Leave one for the autogenerated values, another for webhook secrets, and maybe another for the admin password (which is optionally autogenerated)? I'm encountering the same issue trying to introduce a webhook.gitlab.secret via Helm. And/or use server-side-apply for the autogenerated values?

[GuillaumeTC1]

Hello, I face a related issue on ArgoCD 2.9 managed with Helm Charts. Argo goes out of sync because of required secrets config. If I sync I loose access right away and need to re add manually secrets in Azure.

Am I the only one to encounter this issue ? Do you think it would be possible to fix this without having to ignore argocd-secret sync in argo-cm ?

[jdubs]

I'm also having this same issue with self managed argo. This virtually makes the api/ui not usable, it makes me think that it should fail the health check so that it can be restarted automatically with the new secret.

[fstr]

I ran into the same issue as https://github.com/argoproj/argo-cd/issues/1936#issuecomment-1850161571 It's also simple to reproduce together with the ArgoCD Helm chart and ArgoCD managing itself.

Add a github.webhook.secret:

argo-cd:
  configs:
    secret:
       githubSecret: foobar

Sync it using ArgoCD. Successfully setup. Now your secret will look like this or similar:

apiVersion: v1
data:
  accounts.argoreadonly.tokens: +++++
  admin.password: +++++
  admin.passwordMtime: +++++
  server.secretkey: +++++
  webhook.github.secret: +++++

Now remove the github.webhook.secret by removing the configs: block from your value file again. The argocd-secret will now have an empty data: block when you run helm template because you removed the only value managed by the chart. If you now go into ArgoCD and sync, it will remove all the properties within data property of the argocd-secret. Including all those auto-generated values from the server. After that, ArgoCD stops working. You need to follow the steps from https://github.com/argoproj/argo-cd/issues/1936#issuecomment-1318892424 to resurrect it.

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in https://github.com/argoproj/argo-cd/pull/16262 obsolete.

[MrFreezeex]

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in #16262 obsolete.

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

[fstr]

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

How I got into this topic is also by the need to provide the secret via external-secrets. If the webhook secret would be stored in a separate secret, then we could simply provision that via external-secrets and there's no more need to reference it via the special $<secret-name>.<property> syntax. Your PR will bridge the gap until then though.

[RobinsonZ]

This is also causing an issue for us when argocd-secret is itself managed by argocd, and (as mentioned above) it's compounded by the fact that the argocd-server healthcheck will still succeed even when this error is occurring and making the interface unusable.