'Forbidden' error when creating Immutable secrets

[alexgenon]

Describe the bug I'm trying to create a an ExternalSecret with data coming from a password generator and using a template. As we don't want this password to be refreshed, we set the spec.target.immutable attribute to true. However, whenever I create it, the controller reports the following error

Status:
  Binding:
    Name:  es-5-test
  Conditions:
    Last Transition Time:   2024-06-19T05:40:30Z
    Message:                could not update Secret
    Reason:                 SecretSyncedError
    Status:                 False
    Type:                   Ready
  Refresh Time:             2024-06-19T05:40:30Z
  Synced Resource Version:  1-afdbc4873f1d1bdfd2885d1d7b0e682b
Events:
  Type     Reason        Age                From              Message
  ----     ------        ----               ----              -------
  Normal   Updated       28s                external-secrets  Updated Secret
  Warning  UpdateFailed  1s (x11 over 28s)  external-secrets  Secret "es-5-test" is invalid: data: Forbidden: field is immutable when `immutable` is set

If we set creationPolicy: Orphan then we don't have this issue but the secret is not recreated whenever it is deleted by accident.

To Reproduce Kubernetes version: 1.27.13 ExternalSecrets version: v0.9.16

This is the ExternalSecret manifest

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-5-test
spec:
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: Password
        name: password-generator
  refreshInterval: 0m
  target:
    immutable: true
    name: es-5-test
    template:
      data:
        password: '{{ .password }}'
        username: someone

For the sake of completeness, here's the Password's manifest:

apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: password-generator
spec:
  allowRepeat: true
  length: 16
  noUpper: false
  symbolCharacters: ~!%^&*()_+-={}|[]\<>?,./

Expected behavior Secret is created with immutable: true and ExternalSecret conditions is set to SecretSynced.

Screenshots not applicable

Additional context Add any other context about the problem here.

Many thanks in advance for your support!

[moolen]

Not a bug, this is intentional behaviour. Remove the immutable field and set refreshInterval to 0, that should do.

edit Just re-read. RefreshInterval is 0 :) When you remove the immutable, the password should not change and should only be set once, even after a controller restart. Did you verify that?

[alexgenon]

Hi @moolen, Thanks for your prompt reply!

If we only set the RefreshInterval to 0, there's still a risk that the secret is refreshed if, for example, labels or annotations are updated on the source ExternalSecrets object. In our case, since we intend to use ArgoCD and Helm to deploy those secrets, it could happen that labels or annotations are updated while we don't expect the secret to be refreshed.

We wanted to use the immutable flag to have the guarantee that the secret is not updated, the shouldReconcile function from the external secrets controller would then skip reconciliation.

I don't know if it would make more sense to look at the shouldRefresh function from the external secrets controller and return false if only non "critical" attributes from the ExternalSecret object are updated. But then we need to find another way to manually trigger a secret refresh.

[moolen]

I was able to reproduce the issue, though unfortunately it appears to be a race condition. The .status.condition is empty for some reason and thus shouldReconcile() does not return false to stop reconciliation. It does work 4/5 times on my machine :tm:, so if we resolve that race everything should work as you expect.

[moolen]

Weird stuff happening there :raised_eyebrow:. I've found a workaround. Not pretty, but it seems to work reliably.

See #3608. Can you verify if that works for you?

There's a image available from the PR:

ghcr.io/external-secrets/external-secrets:v0.9.19-38.ga963796b@sha256:1b5c9bc38fc0ae1557772d395a9e2bb0a0a660d7fd84f7817575b3d0044151aa

[alexgenon]

Waw, that was fast, Thanks much! The race condition explains why sometimes it also worked in our environment. I will check with my SRE team to deploy the PR image on our test environment and I'll get back to you with the outcome of my tests.

[alexgenon]

Hi, I forgot to come back and close this issue. Sorry about that! Thanks again for the support