Pulumi wants to replace a `v1.Secret` every run

[thepabloaguilar]

Hello!

I'm using Pulumi to install a Helm chart in a EKS cluster, the first run worked like a charm but everytime after that I try to run pulumi up it wants to replace a secret and the problem is that secret is a protected resource (and should be I guess). This is how I'm installing the chart:

from typing import Any

import pulumi
from pulumi_kubernetes.helm import v3 as helm

from eks.default_cluster.cluster import default_eks_cluster
from eks.default_cluster.provider import k8s_provider


# TODO: Remove this after a Pulumi solution
# See more: https://github.com/pulumi/pulumi-kubernetes/issues/800
def _remove_status(obj: dict[str, Any], opts: pulumi.ResourceOptions) -> None:
    if obj["kind"] == "CustomResourceDefinition":
        del obj["status"]


aws_eks_load_balancer = helm.Chart(
    release_name='aws-load-balancer-controller',
    config=helm.ChartOpts(
        chart='aws-load-balancer-controller',
        namespace='kube-system',
        values={
            'clusterName': default_eks_cluster.name,
            'image': {
                'repository': '602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon/aws-load-balancer-controller',
                'tag': 'v2.4.1',
            },
            'serviceAccount': {
                'create': False,
                'name': 'aws-load-balancer-controller',
            },
        },
        transformations=[_remove_status],
        repo='eks',
        fetch_opts=helm.FetchOpts(
            repo='https://aws.github.io/eks-charts',
        ),
    ),
    opts=pulumi.ResourceOptions(
        parent=default_eks_cluster,
        depends_on=[k8s_provider],
        provider=k8s_provider,
    )
)

pulumi up output:

Previewing update (prod):
     Type                          Name                               Plan     Info
     pulumi:pulumi:Stack           infrastructure-prod                         
     └─ kubernetes:core/v1:Secret  kube-system/aws-load-balancer-tls           1 error
 
Diagnostics:
  kubernetes:core/v1:Secret (kube-system/aws-load-balancer-tls):
    error: unable to replace resource "urn:pulumi:prod::infrastructure::aws:eks/cluster:Cluster$kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`

pulumi up after I unprotected that resource:

$ pulumi up                                                                                                                                                                 
Previewing update (prod):
     Type                                                                                Name                               Plan        Info
     pulumi:pulumi:Stack                                                                 infrastructure-prod                            
     └─ aws:eks:Cluster                                                                  default-eks-cluster                            
        └─ kubernetes:helm.sh/v3:Chart                                                   aws-load-balancer-controller                   
 +-        ├─ kubernetes:core/v1:Secret                                                  kube-system/aws-load-balancer-tls  replace     [diff: ~data]
 ~         ├─ kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration    aws-load-balancer-webhook          update      [diff: ~webhooks]
 ~         └─ kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration  aws-load-balancer-webhook          update      [diff: ~webhooks]
 
Resources:
    ~ 2 to update
    +-1 to replace
    3 changes. 27 unchanged

Issue details

Steps to reproduce

1. Install the aws-load-balancer-controller chart in a EKS Cluster (with pulumi up)
2. Run pulumi up again

Expected: No replacements try Actual: Replacements try

[thepabloaguilar]

I removed and added again the chart using Pulumi, I got this in my second run:

pulumi preview
Previewing update (prod):
     Type                                                                                Name                                                                  Plan        Info
     pulumi:pulumi:Stack                                                                 infrastructure-prod                                                               
     └─ aws:eks:Cluster                                                                  default-eks-cluster                                                               [diff: ~provider]
        └─ kubernetes:helm.sh/v3:Chart                                                   aws-load-balancer-controller                                                      [diff: ~protect]
 +         ├─ kubernetes:elbv2.k8s.aws/v1beta1:IngressClassParams                        alb                                                                   create      
           ├─ kubernetes:core/v1:Service                                                 kube-system/aws-load-balancer-webhook-service                                     [diff: ~protect]
 +-        ├─ kubernetes:core/v1:Secret                                                  kube-system/aws-load-balancer-tls                                     replace     [diff: ~data]
           ├─ kubernetes:networking.k8s.io/v1:IngressClass                               alb                                                                               [diff: ~protect]
           ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition                ingressclassparams.elbv2.k8s.aws                                                  [diff: ~protect]
           ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition                targetgroupbindings.elbv2.k8s.aws                                                 [diff: ~protect]
 ~         ├─ kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration    aws-load-balancer-webhook                                             update      [diff: ~webhooks]
 ~         ├─ kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration  aws-load-balancer-webhook                                             update      [diff: ~webhooks]
           ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole                        aws-load-balancer-controller-role                                                 [diff: ~protect]
           ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding                 aws-load-balancer-controller-rolebinding                                          [diff: ~protect]
           ├─ kubernetes:rbac.authorization.k8s.io/v1:Role                               kube-system/aws-load-balancer-controller-leader-election-role                     [diff: ~protect]
           ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding                        kube-system/aws-load-balancer-controller-leader-election-rolebinding              [diff: ~protect]
           ├─ kubernetes:apps/v1:Deployment                                              kube-system/aws-load-balancer-controller                                          [diff: ~protect]
 -         └─ kubernetes:elbv2.k8s.aws/v1beta1:IngressClassParams                        kube-system/alb                                                       delete

For some reason it's deleting my IngressClassParams named kube-system/alb and creating another with alb as the name!

[viveklak]

For your most recent attempt, I see a ingress class without namespace prefix already so perhaps there was an issue with the naming scheme in your code to install it initially? Could you open a separate ticket when you try it with a fresh install?

For the original issue, that is currently expected and needs better documentation (https://github.com/pulumi/pulumi-kubernetes/issues/1568). We treat secrets as immutable at the moment. We addressed a problem related to configmaps recently and we may have to extend the same to secrets. In this particular case though - perhaps you can consider using the new helm release resource? https://www.pulumi.com/blog/helm-release-resource-for-kubernetes-generally-available/

[thepabloaguilar]

Hey @viveklak thanks for the reply, I've changed to Release and it worked like a charm!

For your most recent attempt, I see a ingress class without namespace prefix already so perhaps there was an issue with the naming scheme in your code to install it initially?

No, it's the same code from the first comment. You can see there the namespace is set correctly: namespace='kube-system',

Could you open a separate ticket when you try it with a fresh install?

On this repo, right?