Use server-side diff by default

[lblackstone]

Server-side diff has been available since 1.16 (September 2019), so it should be pretty widely supported on deployed clusters by now. Most of the implementation work is done already, so the remaining work should mostly be testing, and then swapping server-side diff to be the default.

As of April 30, 2021, EKS and AKS still support 1.15 clusters, but it appears that this support will be ending soon with the recent upstream release of k8s 1.21.

There is one open issue that also needs investigation: https://github.com/pulumi/pulumi-kubernetes/issues/683

Once we're confident in the test coverage, the enableDryRun flag should flip polarity so that it defaults to True.

Edit: The enableDryRun flag has been deprecated in favor of the enableServerSideApply flag in the v3.20.0 release, and will be removed in the next major release.

[lblackstone]

For some additional context, server-side diff fixes the following issues: https://github.com/pulumi/pulumi-kubernetes/issues/924 https://github.com/pulumi/pulumi-kubernetes/issues/694 https://github.com/pulumi/pulumi-kubernetes/issues/641

[lukehoban]

If we move to server-side diff, does that also enable not emitting last-applied-configuration? I believe that was largely only emitted to help with client-side diffs. If so - that might help with https://github.com/pulumi/pulumi-kubernetes/issues/1048 and https://github.com/pulumi/pulumi-kubernetes/issues/1118.

[lblackstone]

If we move to server-side diff, does that also enable not emitting last-applied-configuration?

I believe so. I'm not sure if that has any implications for kubectl integration currently, but my understanding is that the managedFields metadata used by server-side apply provides a more fine-grained alternative to accomplish the same goal.

Edit: Found this article about SSA that explains:

Compared to the last-applied annotation managed by kubectl, Server Side Apply uses a more declarative approach, which tracks a user's field management, rather than a user's last applied state.

kubectl server-side apply keeps the last-applied-configuration annotation up-to-date if you use kubectl apply

[stanislav-zaprudskiy]

If we move to server-side diff, does that also enable not emitting last-applied-configuration? I believe that was largely only emitted to help with client-side diffs. If so - that might help with #1048 and #1118.

Based on a simple test - it looks like the annotation is still being set even with SSA; or something else isn't properly configured/set.

I'm trying to deploy kube-prometheus-stack, but prometheuses CRD fails to be deployed with pulumi complaining on

error: resource prometheuses.monitoring.coreos.com was not successfully created by the Kubernetes API server : CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

I tried deploying it with SSA flag (PULUMI_K8S_ENABLE_DRY_RUN=true / enable_dry_run=True), but running into the same error - now also during preview as well. The code to reproduce:

from pulumi_kubernetes.yaml import ConfigGroup

def remove_status_from_crd(obj):
    if obj["kind"] == "CustomResourceDefinition":
        try:
            del obj["status"]
        except KeyError:
            pass

prometheus_crd = ConfigGroup(
    "prometheus-crd",
    files=[
        "https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml",  # noqa: E501; pylint: disable=C0301
    ],
    transformations=[
        remove_status_from_crd,
    ],
)

It's Docker Desktop kubernetes:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

[lblackstone]

Based on a simple test - it looks like the annotation is still being set even with SSA

Right, the annotation is still present in both modes. I'm hoping to remove it from the SSA mode in the future.

[sathieu]

Have you tried :

kubectl apply --server-side --field-manager=pulumi

Ref: https://kubernetes.io/docs/reference/using-api/server-side-apply/#upgrading-from-client-side-apply-to-server-side-apply

[KaoruDev]

Is there documentation on how I would go about migrating to having this flag turned on? When I flip this to true and run pulumi update I get the following error:

 panic: fatal: An assertion has failed: path: [spec nodeSets 0 podTemplate metadata creationTimestamp]  |  v: <nil>  | old: <nil>  |  oldInput: <nil>
    goroutine 657 [running]:
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/failfast.go:23
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0x0, {0x22c6cfa, 0x8}, {0xc00124d768, 0x37eff60, 0xc00045eea0})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/assert.go:33 +0xf4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001eea800, 0x0, 0x8}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2913 +0x188
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc001eea800, {0xc001eea800, 0x5, 0x8}, 0x203000, 0x203000, 0xc001eea800, 0x3, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3024 +0x65a
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001eea800, 0x8, 0x8}, {0x1f39380, 0xc002cc9ad0}, {0x1f39380, 0xc002e7a9f0}, {0x1f39380, 0xc002e68ab0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc001ec1d80, {0xc001ec1d80, 0x4, 0x4}, 0x2, 0xffffffffffffffff, 0x40, 0x202b280, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001ec1d80, 0x8, 0x4}, {0x1f39380, 0xc002cc9aa0}, {0x1f39380, 0xc002e7a9c0}, {0x1f39380, 0xc002e68a80}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc001ec1d80, {0xc001ec1d80, 0x3, 0x4}, 0x203000, 0x203000, 0xc001ec1d80, 0x57780f, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001ec1d80, 0x8, 0x4}, {0x1f39380, 0xc002cc9a40}, {0x1f39380, 0xc002e7a960}, {0x1f39380, 0xc002e68a20}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchArrayToDiff(0x0, {0xc002061b60, 0x2, 0x2}, {0xc002101560, 0x1, 0xc00124e920}, {0xc00024e9c0, 0x1, 0x1}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3046 +0x23e
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc002061b60, 0x74b89a4310045424, 0x2}, {0x1e014a0, 0xc002a3a3c0}, {0x1e014a0, 0xc00293c3c0}, {0x1e014a0, 0xc0028b6060}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2943 +0x405
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc002101640, {0xc002101640, 0x1, 0x1}, 0xc00124ee80, 0x40d0b4, 0x0, 0x7f9011329ab0, 0x0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc002101640, 0x1e573c0, 0x1}, {0x1f39380, 0xc002cc9980}, {0x1f39380, 0xc002e690e0}, {0x1f39380, 0xc002e68930}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0x2, {0x0, 0x0, 0x0}, 0x20, 0x1e573c0, 0x10000c001ef8001, 0xffffffffffffffff, 0x0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff(0xc001f20d20, 0xc002e69020, 0x1f39380, 0xc002cae060, {0xc002061ae0, 0x2, 0x2})
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2841 +0x105
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider).Diff(0xc0004cbe00, {0x26238a8, 0xc002ade330}, 0xc00138af50)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1554 +0x1cf9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x26238a8, 0xc002ade330}, {0x2151fe0, 0xc00138af50})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2604 +0x78
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x26238a8, 0xc002906660}, {0x2151fe0, 0xc00138af50}, 0xc0007291a0, 0xc0028da4e0)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x406
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x21f1ee0, 0xc0004cbe00}, {0x26238a8, 0xc002906660}, 0xc002b94300, 0xc000517000)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2606 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004dd340, {0x26643f8, 0xc00058bd40}, 0xc001e1b440, 0xc000a2cc30, 0x3a9ac28, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1282 +0xccf
    google.golang.org/grpc.(*Server).handleStream(0xc0004dd340, {0x26643f8, 0xc00058bd40}, 0xc001e1b440, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1619 +0xa2a
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:921 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:919 +0x294

[RossComputerGuy]

If we move to server-side diff, does that also enable not emitting last-applied-configuration? I believe that was largely only emitted to help with client-side diffs. If so - that might help with #1048 and #1118.

Based on a simple test - it looks like the annotation is still being set even with SSA; or something else isn't properly configured/set.

I'm trying to deploy kube-prometheus-stack, but prometheuses CRD fails to be deployed with pulumi complaining on

error: resource prometheuses.monitoring.coreos.com was not successfully created by the Kubernetes API server : CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

I tried deploying it with SSA flag (PULUMI_K8S_ENABLE_DRY_RUN=true / enable_dry_run=True), but running into the same error - now also during preview as well. The code to reproduce:

from pulumi_kubernetes.yaml import ConfigGroup

def remove_status_from_crd(obj):
    if obj["kind"] == "CustomResourceDefinition":
        try:
            del obj["status"]
        except KeyError:
            pass

prometheus_crd = ConfigGroup(
    "prometheus-crd",
    files=[
        "https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml",  # noqa: E501; pylint: disable=C0301
    ],
    transformations=[
        remove_status_from_crd,
    ],
)

It's Docker Desktop kubernetes:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

I'm getting this exact issue with my local development using Minikube except I don't use Python, I use NodeJS. Have you found a solution because this is preventing my stuff from working?

[lblackstone]

Update: Server-side diff/apply can now be enabled with the enableServerSideApply mode available in the v3.20.1 release. This is expected to become the default provider behavior as part of the v4.0.0 release.

[lblackstone]

SSA mode was enabled in the v3.22.0 release

[mortaelth]

is it to be reopened? The default was reverted: https://github.com/pulumi/pulumi-kubernetes/pull/2206

[lblackstone]

We rolled back this default in v3.22.1, and have decided to wait for the v4.0.0 release to enable it by default.