Update Secret Store CSI add-on

[madchap]

Description

#473 got closed for inactivity.

The CSI Secret Store version from blueprint-addons is 0.3.11. However, the latest version from the sig is at 1.5.x. Eventually, the addon may stop working as software evolves.

Upon creation of secrets store csi and aws provider, DS and pods are deployed and running, yet the terraform informs of:

Failed to execute "/opt/homebrew/bin/tofu apply -auto-approve" in .
  ╷
  │ Error: 8 errors occurred:
  │     * serviceaccounts "secrets-store-csi-driver" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasses-admin-role" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasses-viewer-role" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasspodstatuses-viewer-role" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasses-role" already exists
  │     * clusterrolebindings.rbac.authorization.k8s.io "secretproviderclasses-rolebinding" already exists
  │     * daemonsets.apps "secrets-store-csi-driver-provider-aws" already exists
  │     * csidrivers.storage.k8s.io "secrets-store.csi.k8s.io" already exists
  │
  │
  │
  │   with module.eks_blueprints_addons.module.secrets_store_csi_driver_provider_aws.helm_release.this[0],
  │   on .terraform/modules/eks_blueprints_addons.secrets_store_csi_driver_provider_aws/main.tf line 9, in resource "helm_release" "this":
  │    9: resource "helm_release" "this" {
  │
  ╵

  exit status 1

• [x] ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

[DONE]

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]:

• Terraform version:

• Provider version(s):

OpenTofu v1.10.5
on darwin_arm64
+ provider registry.opentofu.org/go-gandi/gandi v2.3.0
+ provider registry.opentofu.org/hashicorp/aws v5.100.0
+ provider registry.opentofu.org/hashicorp/cloudinit v2.3.7
+ provider registry.opentofu.org/hashicorp/helm v2.17.0
+ provider registry.opentofu.org/hashicorp/kubernetes v2.38.0
+ provider registry.opentofu.org/hashicorp/null v3.2.4
+ provider registry.opentofu.org/hashicorp/random v3.7.2
+ provider registry.opentofu.org/hashicorp/time v0.13.1
+ provider registry.opentofu.org/hashicorp/tls v4.1.0

Reproduction Code [Required]

Steps to reproduce the behavior. Force the version upgrade.

configured as such:

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0" #ensure to update this to the latest/desired version

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_metrics_server                        = true
  enable_secrets_store_csi_driver              = true
  enable_secrets_store_csi_driver_provider_aws = true

  [... other modules config ...]

  secrets_store_csi_driver_provider_aws = {
    # https://github.com/aws/secrets-store-csi-driver-provider-aws/releases
    # pod identity support added in 0.3.11, default version in eks_blueprint_addons is older.
    chart_version : 2.1.1
  }
  secrets_store_csi_driver = {
    chart_version = 1.5.3,
    set : [
      {
        # allows to create k8s secrets from secrets CSI driver
        name  = "syncSecret.enabled"
        value = "true"
      }
    ]
  }
}

I am using workspaces.

Yes, I have cleared my cache.

1. Set secrets store to false to fully uninstall the CSIs.
2. Observe all resources are gone
3. Reset to true.
4. Observe all resources are here, DS and pods running.
5. Observe tofu still outputs an error.

Expected behaviour

tofu does not output any errors.

Actual behaviour

11:36:42.261 ERROR  tofu invocation failed in .
11:36:42.262 ERROR  error processing error handling rules: error occurred:

* Failed to execute "/opt/homebrew/bin/tofu apply -auto-approve" in .
  ╷
  │ Error: 8 errors occurred:
  │     * serviceaccounts "secrets-store-csi-driver" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasses-admin-role" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasses-viewer-role" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasspodstatuses-viewer-role" already exists
  │     * clusterroles.rbac.authorization.k8s.io "secretproviderclasses-role" already exists
  │     * clusterrolebindings.rbac.authorization.k8s.io "secretproviderclasses-rolebinding" already exists
  │     * daemonsets.apps "secrets-store-csi-driver-provider-aws" already exists
  │     * csidrivers.storage.k8s.io "secrets-store.csi.k8s.io" already exists
  │
  │
  │
  │   with module.eks_blueprints_addons.module.secrets_store_csi_driver_provider_aws.helm_release.this[0],
  │   on .terraform/modules/eks_blueprints_addons.secrets_store_csi_driver_provider_aws/main.tf line 9, in resource "helm_release" "this":
  │    9: resource "helm_release" "this" {

On a possible subsequent invocation, secrets-store-csi-driver-provider-aws will continuously be failed and re-installed.

OpenTofu will perform the following actions:
11:34:00.414 STDOUT tofu:   # module.eks_blueprints_addons.module.secrets_store_csi_driver_provider_aws.helm_release.this[0] is tainted, so it must be replaced
11:34:00.414 STDOUT tofu: -/+ resource "helm_release" "this" {
11:34:00.414 STDOUT tofu:       ~ id                         = "secrets-store-csi-driver-provider-aws" -> (known after apply)
11:34:00.414 STDOUT tofu:       + manifest                   = (known after apply)
11:34:00.414 STDOUT tofu:       ~ metadata                   = [
11:34:00.414 STDOUT tofu:           - {
11:34:00.414 STDOUT tofu:               - app_version    = ""
11:34:00.414 STDOUT tofu:               - chart          = "secrets-store-csi-driver-provider-aws"
11:34:00.415 STDOUT tofu:               - first_deployed = 1759311118
11:34:00.415 STDOUT tofu:               - last_deployed  = 1759311118
11:34:00.415 STDOUT tofu:               - name           = "secrets-store-csi-driver-provider-aws"
11:34:00.415 STDOUT tofu:               - namespace      = "kube-system"
11:34:00.415 STDOUT tofu:               - notes          = <<-EOT
11:34:00.415 STDOUT tofu:                     The Secrets Store CSI Driver is getting deployed to your cluster.
11:34:00.415 STDOUT tofu:
11:34:00.415 STDOUT tofu:                     To verify that Secrets Store CSI Driver has started, run:
11:34:00.415 STDOUT tofu:
11:34:00.415 STDOUT tofu:                       kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
11:34:00.415 STDOUT tofu:
11:34:00.415 STDOUT tofu:                     Now you can follow these steps https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html
11:34:00.415 STDOUT tofu:                     to create a SecretProviderClass resource, and a deployment using the SecretProviderClass.
11:34:00.415 STDOUT tofu:                 EOT
11:34:00.415 STDOUT tofu:               - revision       = 1
11:34:00.415 STDOUT tofu:               - values         = jsonencode({})
11:34:00.415 STDOUT tofu:               - version        = "2.1.1"
11:34:00.415 STDOUT tofu:             },
11:34:00.415 STDOUT tofu:         ] -> (known after apply)
11:34:00.415 STDOUT tofu:         name                       = "secrets-store-csi-driver-provider-aws"
11:34:00.415 STDOUT tofu:       ~ status                     = "failed" -> "deployed"
11:34:00.415 STDOUT tofu:         # (27 unchanged attributes hidden)
11:34:00.415 STDOUT tofu:     }
11:34:00.415 STDOUT tofu: Plan: 1 to add, 0 to change, 1 to destroy.

leading to

* Failed to execute "/opt/homebrew/bin/tofu apply" in .
  ╷
  │ Error: 1 error occurred:
  │     * daemonsets.apps "secrets-store-csi-driver-provider-aws" already exists
  │
  │
  │
  │   with module.eks_blueprints_addons.module.secrets_store_csi_driver_provider_aws.helm_release.this[0],
  │   on .terraform/modules/eks_blueprints_addons.secrets_store_csi_driver_provider_aws/main.tf line 9, in resource "helm_release" "this":
  │    9: resource "helm_release" "this" {
  f│
 f ╵
f
  exit status 1

Terminal Output Screenshot(s)

Additional context

All seems OK, however:

$ k get pods -l app.kubernetes.io/instance=secrets-store-csi-driver -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
secrets-store-csi-driver-48s5r   3/3     Running   0          12m
secrets-store-csi-driver-l757d   3/3     Running   0          12m
secrets-store-csi-driver-ncs4k   3/3     Running   0          12m
secrets-store-csi-driver-npf6s   3/3     Running   0          12m
secrets-store-csi-driver-rwwgg   3/3     Running   0          12m
secrets-store-csi-driver-wk8wr   3/3     Running   0          12m
secrets-store-csi-driver-wx7jr   3/3     Running   0          12m

$ k get pods -l app.kubernetes.io/instance=secrets-store-csi-driver-provider-aws -n kube-system
NAME                                          READY   STATUS    RESTARTS   AGE
secrets-store-csi-driver-provider-aws-6ztc7   1/1     Running   0          2m36s
secrets-store-csi-driver-provider-aws-8klzs   1/1     Running   0          2m36s
secrets-store-csi-driver-provider-aws-clddk   1/1     Running   0          2m36s
secrets-store-csi-driver-provider-aws-ggn94   1/1     Running   0          2m36s
secrets-store-csi-driver-provider-aws-rzwn2   1/1     Running   0          2m36s

The only option is to rollback to the initial 0.3.11 version offered with blueprints-addons.