replaceOnChange doesn't work for immutable Fields

[bernhardloos]

What happened?

Making changes to the storage class field of a PersistentVolumeClaim fails with an error during preview:

error: Preview failed: resource ... was not successfully created by the Kubernetes API server : PersistentVolumeClaim "..." is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
      core.PersistentVolumeClaimSpec{
        ... // 2 identical fields
        Resources:        {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}},
        VolumeName:       "...",
    -   StorageClassName: &"no-provisioner",
    +   StorageClassName: &"no-provisionerf",
        VolumeMode:       &"Filesystem",
        DataSource:       nil,
        DataSourceRef:    nil,
      }

Setting replaceOnChange doesn't seem to have any effect, no matter what I put there. I suspect the provider sends the data to the API server for server side apply before the replaceOnChange option is processed. replaceOnChange works if use it for example for annotations.

Example

any PVC should exhibit the behavior: https://www.pulumi.com/registry/packages/kubernetes/api-docs/core/v1/persistentvolumeclaim/

Output of pulumi about

CLI          
Version      3.81.0
Go Version   go1.21.0
Go Compiler  gc

Plugins
NAME             VERSION
kubernetes       4.1.1
kubernetes_crds  4.1.1
python           unknown
random           4.13.2
tls              4.10.0
vault            5.15.0

Host     
OS       fedora
Version  38
Arch     x86_64

This project is written in python: executable='/usr/bin/python3' version='3.11.4


Found no pending operations associated with main

Backend        
Name           lap
URL            file://.
User           bl
Organizations  

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

Hi @bernhardloos, thanks for reporting this issue, and apologies you're facing it. Your hypothesis is on the right track. If there is a diff detected, Pulumi attempts to also do a server side apply dry run. Since an update on immutable fields is prohibited, the dry run apply will fail which results in what you observed. I'll dig into what we can do to make this experience better, but I believe the fix will probably require us to NOT do a dry-run if replaceOnChange is specified.

For now, a workaround to this issue would be to skip preview by running pulumi up --skip-preview to unblock you from updating PVCs.

[bernhardloos]

You could try to do something clever and do a dry-run with a different (random) name if a replace is detected but I'm not sure if this will break in some corner cases.

[chasinglogic]

This is affecting us with our database migrations (which we currently create as Jobs).

I'd be interested in fixing this if there was a clear direction on what solution has buy in from the maintainers.

[mortaelth]

In my opinion this is general issue with the server-side apply. It is affecting any resource which is named using the metadata.name AND when it receives a change which generates a replace of the resource. Kubernetes job is the most common example in our case. When the job receives update of the container image, it fails with "field is immutable" error

[mortaelth]

This is affecting us with our database migrations (which we currently create as Jobs).

I'd be interested in fixing this if there was a clear direction on what solution has buy in from the maintainers.

hello @chasinglogic, did you find any other solution than to suppress the server-side apply for the time-being? For us it is a better workaround than giving up on the preview (as with the --skip-preview it works correctly)

[mortaelth]

I have reproduced the issue without pulumi using just the kubectl:

kubectl patch job job-name --patch-file patch.json --dry-run='server'

patch.json contains something like: '{ "spec": { "template": { "spec": { "containers": [ ..container object with changed image label.. ] } } } }

Am I missing something here? Is there a trick Pulumi is doing to make SSA work? Otherwise it is not possible to maintain jobs from pulumi while having the SSA enabled.

[mortaelth]

anyone knows any solution? This is blocker for us in enabling the server-side apply

[giovannicandido]

I'm having the same problem but with deployments:

     Type                                             Name                             Plan        Info
     pulumi:pulumi:Stack                              guess-game-ui-hom                            1 error; 6 messages
     └─ k8s-app-deployment:kubernetes:app-deployment  guess-game-ui
 ~      ├─ kubernetes:cert-manager.io/v1:Certificate  guess-game-ui-certificate        update      [diff: ~metadata,provider]
 ~      ├─ kubernetes:traefik.io/v1alpha1:Middleware  guess-game-ui-redirect-to-https  update      [diff: ~metadata,provider]
 +-     └─ kubernetes:apps/v1:Deployment              guess-game-ui                    replace     [diff: ~metadata,provider,spec]; 1 error
Diagnostics:
  kubernetes:apps/v1:Deployment (guess-game-ui):
    error: Preview failed: resource "urn:pulumi:hom::guess-game-ui::k8s-app-deployment:kubernetes:app-deployment$kubernetes:apps/v1:Deployment::guess-game-ui" was not successfully 
created by the Kubernetes API server : Deployment.apps "guess-game-ui" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/i
nstance":"guess-game-ui", "app.kubernetes.io/name":"guess-game-ui", "app.kubernetes.io/version":"v20240501.1"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable