wireguard-wg-generated secrets exists every sync run of ArgoCD causing it to get stuck

[syphernl]

We're running this chart with ArgoCD. Every time it syncs it runs the job to generate the secret, this fails with

error: failed to create secret secrets "wireguard-wg-generated" already exists

and causes ArgoCD to get stuck in:

waiting for completion of hook batch/Job/wireguard-wg-gen and 6 more hooks

In order to get it to proceed we need to remove the secret, but with every new sync this is breaking again.

[bryopsida]

Looks to be related to this: https://github.com/argoproj/argo-cd/issues/7536

Normally the hooks only run on initial install but with Argo pre-install hooks are run on every sync.

Have you tried setting the secretName to the created secret? https://github.com/bryopsida/wireguard-chart/blob/5929db46ae0a995fbc7433630b8f9eeb41901979/helm/wireguard/values.yaml#L70

That would disable the render of the hook https://github.com/bryopsida/wireguard-chart/blob/5929db46ae0a995fbc7433630b8f9eeb41901979/helm/wireguard/templates/privatekey-gen-job.yaml#L1

[bryopsida]

If that doesn't work for you I can adjust the job to check if the secret already exists and exit without modifying it if it does.

[syphernl]

Thanks for your quick reply @bryopsida !

Setting secretName: wireguard-wg-generated works, but this is probably more a workaround than an actual solution as it would require you to omit the value for the initial deploy and and update it later on. It would be easier/better if the job could check if the secret exists and skip it if it does as it would simplify the deploy process (making it only 1 step rather than 2).

[bryopsida]

@syphernl I've started working on this over here: https://github.com/bryopsida/k8s-wireguard-mgr when I roll it into the chart it will be an opt in value toggle to replace the current hook behavior which uses a configmap shell script: https://github.com/bryopsida/wireguard-chart/blob/main/helm/wireguard/scripts/gen-key.sh

There is one variation to the behavior from what I had previously mentioned. Instead of checking/fetching the secret it will attempt to create the secret and just inspect the error response and exit cleanly if the error reason is AlreadyExists.

The reason for this is I do not want to expand the role bindings to give the job access to fetch the secret, currently it only has permission to create.

https://github.com/bryopsida/wireguard-chart/blob/5929db46ae0a995fbc7433630b8f9eeb41901979/helm/wireguard/templates/privatekey-gen-job.yaml#L22

[dro-sh]

Is there any updates?

[bryopsida]

I started on a PR here: https://github.com/bryopsida/wireguard-chart/pull/51 but need to do some more testing etc before integrating it.

It does seem like there a few ways to work with argo with the current version of the chart

1. Use a secrets provider such as external-secrets and have it manage the secret, at that point you can disable the secret generation hook/job just by setting the secret name: https://github.com/bryopsida/wireguard-chart/blob/146ea71033d432250ab1c716392267d426a965b8/helm/wireguard/templates/privatekey-gen-job.yaml#L1 or use another means to pre-create the secret.
2. The script/hook can be customized using: https://github.com/bryopsida/wireguard-chart/blob/146ea71033d432250ab1c716392267d426a965b8/helm/wireguard/templates/privatekey-gen-job.yaml#L76 and https://github.com/bryopsida/wireguard-chart/blob/146ea71033d432250ab1c716392267d426a965b8/helm/wireguard/templates/privatekey-gen-job.yaml#L154, the hook image has wg + kubectl included.

[bryopsida]

@dro-sh @syphernl this should be resolved in version 0.25.0

If you enable this flag

keygenJob.useWireguardManager = true

The key gen job will succeed even if the key already exists.

[dro-sh]

Thanks! It works fine!