Can't supply kubeconfig path using KUBE_CONFIG_PATH env variable

[v01t]

Hello,

I faced situation when I can't dynamically define path to kubeconfig in flux provider without using extra (bridge) variable:

provider "flux" {
  kubernetes = {
    config_path = var.kubeconfig
  }
  git = {
    url = "ssh://${var.github_ssh_user}@${var.github_url}/${var.github_org}/${var.github_repository}.git"
    branch = var.github_branch
    ssh = {
      username = var.github_ssh_user
      private_key = tls_private_key.flux.private_key_pem
    }
  }
}

according to documentation config_path for kubernetes attribute could be supplied using KUBE_CONFIG_PATH env variable, but it doesn't work together with git attribute:

provider "flux" {
  kubernetes = {
  }
  git = {
    url = "ssh://${var.github_ssh_user}@${var.github_url}/${var.github_org}/${var.github_repository}.git"
    branch = var.github_branch
    ssh = {
      username = var.github_ssh_user
      private_key = tls_private_key.flux.private_key_pem
    }
  }
}

lead to following error:

│ Error: Kubernetes Client
│
│   with flux_bootstrap_git.main,
│   on flux.tf line 14, in resource "flux_bootstrap_git" "main":
│   14: resource "flux_bootstrap_git" "main" {
│
│ invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

same for

  kubernetes = {
    config_path = ""
  }

How I could supply path to kubeconfig without bridging value like: $TF_VAR_kubeconfig > var.kubeconfig > kubernetes.config_path and make it work directly: $KUBE_CONFIG_PATH > kubernetes.config_path in my case?

Version: fluxcd/flux v0.25.3

Thanks

[phillebaba]

The provider does not attempt to fetch attribute values from environment variables. The reason for this is that there is currently no good support to implement this in the new provider SDK. The docs are incorrect, I will create a PR that fixes it, sorry for the confusion.

[viceice]

why not doing something like the kubernetes provider?

https://github.com/hashicorp/terraform-provider-kubernetes/blob/5bc27bc9b9e2831cb43d838bf9017f14130dca65/kubernetes/provider.go#L90-L96

[viceice]

			"config_path": {
				Type:          schema.TypeString,
				Optional:      true,
				DefaultFunc:   schema.EnvDefaultFunc("KUBE_CONFIG_PATH", nil),
				Description:   "Path to the kube config file. Can be set with KUBE_CONFIG_PATH.",
				ConflictsWith: []string{"config_paths"},
			},

https://github.com/fluxcd/terraform-provider-flux/blob/9c8106ca62882ee7c4da12ee09f7de227cc30f28/internal/provider/provider.go#L147-L150

[networkhermit]

Not only does the kubernetes provider, the helm provider also supports setting a set of common kubernetes arguments via environment variables:

KUBE_CONFIG_PATH
KUBE_CLIENT_CERT_DATA
KUBE_CLIENT_KEY_DATA
KUBE_CLUSTER_CA_CERT_DATA
KUBE_HOST
...

I think it's better if flux could align with the popular kubernetes and helm provider, as that improves friction.

[swade1987]

@v01t @networkhermit

Is this missing functionality blocking your ability to use the provider?

The reason I ask is that this provider uses github.com/hashicorp/terraform-plugin-framework

The kubernetes and helm providers use github.com/hashicorp/terraform-plugin-sdk which is seen as the old way.

See snippet from the plugin-sdk README below:

For new provider development it is recommended to investigate terraform-plugin-framework, which is a reimagined provider SDK that supports additional capabilities. Refer to the Which SDK Should I Use? documentation for more information about differences between SDKs.

[networkhermit]

Edited:

~~I can't reproduce this issue in my testing k3s cluster and I'm still trying to understand the reason behind it. It might be environment specific or perhaps unintentionally resolved by some dependency bump. If anyone can still reproduce this issue, please comment below.~~

I managed to verify that #506 added support for KUBE_CONFIG_PATH and KUBE_CONFIG_PATH support since v1.0.0.

But the other options such as KUBE_CLIENT_CERT_DATA/KUBE_CLIENT_KEY_DATA/KUBE_CLUSTER_CA_CERT_DATA/KUBE_HOST still not supported.

---

I don't think it's a blocking issue, but a good enhancement worth considering:

Potential terraform-provider-flux users are highly likely to already using kubernetes and helm provider.

So they might got the following environment variables and terraform provider before they integrate terraform-provider-flux:

The following .env demo is based on 1password-cli, but they can also be injected by CI or IaC platform such as Terraform Cloud:

KUBE_CLIENT_CERT_DATA=op://${VAULT}/kubernetes/${SECTION}/KUBE_CLIENT_CERT_DATA
KUBE_CLIENT_KEY_DATA=op://${VAULT}/kubernetes/${SECTION}/KUBE_CLIENT_KEY_DATA
KUBE_CLUSTER_CA_CERT_DATA=op://${VAULT}/kubernetes/${SECTION}/KUBE_CLUSTER_CA_CERT_DATA
KUBE_HOST=op://${VAULT}/kubernetes/${SECTION}/KUBE_HOST

Or they can provide a dev credential on the local machine:

KUBE_CONFIG_PATH=~/.kube/config

And the terraform provider config is the same:

provider "kubernetes" {}

provider "helm" {
  kubernetes {}
}

As you see, the credentials config of kubernetes and helm can be dynamically set and they are decoupled from the terraform code.

OK. Now the users want to integrate terraform-provider-flux into their codebase or IDP:

1. Add extra variables as bridge:

variable "KUBE_CLIENT_CERT_DATA" {
  nullable = false
  type     = string
}

variable "KUBE_CLIENT_KEY_DATA" {
  nullable  = false
  sensitive = true
  type      = string
}

variable "KUBE_CLUSTER_CA_CERT_DATA" {
  nullable = false
  type     = string
}

variable "KUBE_HOST" {
  nullable = false
  type     = string
}

2. Configure flux to use the bridged variables:

provider "flux" {
  kubernetes = {
    client_certificate     = var.KUBE_CLIENT_CERT_DATA
    client_key             = var.KUBE_CLIENT_KEY_DATA
    cluster_ca_certificate = var.KUBE_CLUSTER_CA_CERT_DATA
    host                   = var.KUBE_HOST
  }
  git = {}
}

3. Add a new set of environment variables to .env or IDP or Terraform Cloud:

TF_VAR_KUBE_CLIENT_CERT_DATA=op://${VAULT}/kubernetes/${SECTION}/KUBE_CLIENT_CERT_DATA
TF_VAR_KUBE_CLIENT_KEY_DATA=op://${VAULT}/kubernetes/${SECTION}/KUBE_CLIENT_KEY_DATA
TF_VAR_KUBE_CLUSTER_CA_CERT_DATA=op://${VAULT}/kubernetes/${SECTION}/KUBE_CLUSTER_CA_CERT_DATA
TF_VAR_KUBE_HOST=op://${VAULT}/kubernetes/${SECTION}/KUBE_HOST

These are so much boilerplate code, and please note that now the authentication method is locked, so if users want to use file based credential such as kubeconfig or change some authentation option, they must change the flux terraform provider code.

I don't think terraform-plugin-framework couldn't use environment variables as provider credentials. They even provide an example in the official document.

And terraform-provider-flux seems already using this mechanism:

https://github.com/fluxcd/terraform-provider-flux/blob/6042c15d8328d37cd99823af9cd4e166d36e6b5a/internal/provider/provider.go#L360-L364

Common set of environment variables of kubernetes and helm provider can be found in here and here. If terraform-provider-flux already support these variables, add an option to sourcing them from these envVars is a big improvement to the UX.

P.S.

For real world terraform users, they might separate their terraform code to modules and workspaces, and when they do the above mentioned boilerplate code would magnified.

[swade1987]

@networkhermit I am going to track this enhancement in https://github.com/fluxcd/terraform-provider-flux/issues/655 and aim to have it released either in the upcoming 1.3.0 release or a fast follow in a 1.3.1 release.

[networkhermit]

@networkhermit I am going to track this enhancement in #655 and aim to have it released either in the upcoming 1.3.0 release or a fast follow in a 1.3.1 release.

@swade1987 ~Thanks. I want to remind you that I can't reproduce this issue in my testing k3s cluster in case you missed my edited comment.~

~I just downgraded my provider version to fluxcd/flux v0.25.3 as the author once used but got no vein trying to reproduce the error. It bugs me.~

~I'm using OpenTofu v1.6.2 and the cluster is already bootstrapped (I can't destroy the cluster as it have some services on it). I checked the terraform plan and there is no error. Maybe this error would happen in the actual terraform apply phase but currently I don't have any brand new cluster so I can't help verify it.~

~If you would work on this enhancement, please don't forget to reproduce it first.~

Thanks again for the attention.

Edited:

I managed to verify that #506 added support for KUBE_CONFIG_PATH and KUBE_CONFIG_PATH support since v1.0.0.

But the other options such as KUBE_CLIENT_CERT_DATA/KUBE_CLIENT_KEY_DATA/KUBE_CLUSTER_CA_CERT_DATA/KUBE_HOST still not supported.