Setting the KUBECONFIG environment variable updates the 'kubeconfig' parameter of the provider

[serhii-samoilenko]

What happened?

I'm writing Pulumi stacks using C#. When explicitly defining Kubernetes providers, the calculated state depends on the KUBECONFIG environment variable. If the variable is set, Pulumi will store its value to a state as a kubeconfig parameter for a kubernetes provider.

This behavior causes issues during the refresh phase. The stored kubeconfig value points to the path of the kubeconfig file from the original environment. If the refresh operation is executed on another machine where the kubeconfig file is located at a different path, the operation fails.

I believe that the kubeconfig localion should only be stored in the state if it's explicitly defined in the provider code, rather than relying on environment variables. Additionally, I could not reproduce this issue with YAML-defined stacks, so the behavior is not consistent between languages.

Example

On a Linux/MacOS:

1. Create a new project with pulumi new kubernetes-csharp
2. Update Program.cs to contain just a one-liner:

return await Pulumi.Deployment.RunAsync(() => new Pulumi.Kubernetes.Provider("k8s"));

3. Make sure the KUBECONFIG is not set:

unset KUBECONFIG

4. Deploy the stack with pulumi up:

     Type                            Name                  Status              
 +   pulumi:pulumi:Stack             pulmi-kubeconfig-dev  created (2s)        
 +   └─ pulumi:providers:kubernetes  k8s                   created (0.29s)     

Resources:
    + 2 created

5. Set some KUBECONFIG value:

export KUBECONFIG=~/.kube/config

6. Update the stack with pulumi up. It will try to update the k8s provider with a new kubeconfig value:

     Type                            Name                  Plan       Info
     pulumi:pulumi:Stack             pulmi-kubeconfig-dev             
 ~   └─ pulumi:providers:kubernetes  k8s                   update     [diff: +kubeconfig]

Resources:
    ~ 1 to update
    1 unchanged

Output of pulumi about

CLI          
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME        VERSION
language  dotnet      unknown
resource  kubernetes  4.18.1

Host     
OS       darwin
Version  14.6.1
Arch     x86_64

This project is written in dotnet: executable='/usr/local/share/dotnet/dotnet' version='8.0.204'

Current Stack: my_org/pulmi-kubeconfig/dev

TYPE                         URN
pulumi:pulumi:Stack          urn:pulumi:dev::pulmi-kubeconfig::pulumi:pulumi:Stack::pulmi-kubeconfig-dev
pulumi:providers:kubernetes  urn:pulumi:dev::pulmi-kubeconfig::pulumi:providers:kubernetes::k8s

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/serhii_samoilenko
User           serhii_samoilenko
Organizations  serhii_samoilenko
Token type     personal

Dependencies:
NAME               VERSION
Pulumi             3.67.1
Pulumi.Kubernetes  4.18.1

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

Thanks for bringing up this issue. On a cursory glance, this may be something we choose to fix in the next major version release of this provider since it is a pretty big behavior change. I will bring this up in our maintainer's sync meeting today for further consideration.

[EronWright]

Related: https://github.com/pulumi/pulumi-kubernetes/pull/1166

[rquitales]

Further discussions on this subject reveals that this was done intentionally in the past to support obtaining the default kubeconfig file. This was done when the provider did not support loading ambient kubeconfig in the provider logic, and so, was done at the SDK level. This is the reason why YAML behaves differently, since it doesn't have the SDK overrides for eagerly setting the kubeconfig property. We have added this to our backlog to resolve for the next major version release of this provider.

[serhii-samoilenko]

Thanks for paying attention to this. I looked at the code and understand there is no simple fix for this, so it's okay to fix it in the major release. Meanwhile, there is a simple workaround for this issue. Since the value is being set in the Pulumi.Kubernetes.ProviderArgs constructor, one may override it to null right after construction:

    var provider = new Provider("k8s", new ProviderArgs
    {
        KubeConfig = null
    });

[aureq]

related to #3608