`kubernetes.yaml.*` produces a `TypeError: Cannot read properties of undefined`

[omninonsense]

What happened?

If the default kubernetes provider is disabled, the kubernetes.yaml "submodule" doesn't work as expected. I'm not entirely sure if this is related to disabling the default provider, but I assume it is.

Steps to reproduce

I've created a small reproduction in omninonsense/pulumi-repro.

But the general steps to reproduce it should probably be:

• Disable the default kubernetes provider (dunno if this is a red herring)
• Try to use kubernetes.yaml.ConfigFile (or others) with an explicit opts.provider

Expected Behavior

I expected it to apply the manifests using the provider passed via options.

Actual Behavior

I received the following error:

Diagnostics:
pulumi:pulumi:Stack (pulumi-repro-dev):
	error: TypeError: Cannot read properties of undefined (reading 'map')
			at /Users/nino/Documents/silicoai/pulumi-repro/node_modules/@pulumi/yaml/yaml.ts:2993:14
			at processTicksAndRejections (node:internal/process/task_queues:95:5)

Versions used

pulumi about
CLI
Version      3.34.1
Go Version   go1.18.3
Go Compiler  gc

Plugins
NAME        VERSION
aws         5.8.0
command     0.3.0
docker      3.2.0
eks         0.40.0
kubernetes  3.19.3
nodejs      unknown

Host
OS       darwin
Version  12.4
Arch     arm64

This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v18.3.0'

Backend
Name           pulumi.com
URL            https://app.pulumi.com/omninonsense
User           omninonsense
Organizations  omninonsense, EpicGames, silico

Dependencies:
NAME                VERSION
@pulumi/aws         5.8.0
@pulumi/awsx        0.40.0
@pulumi/eks         0.40.0
@pulumi/kubernetes  3.19.3
@pulumi/pulumi      3.34.1
@types/node         14.18.21

Pulumi locates its logs in /var/folders/xm/qbl84xzs55g88fvzdn63nkdr0000gn/T/ by default
warning: Failed to get information about the current stack: No current stack

Additional context

There's a hardcoded AWS providerCredentialOpts.profileName (set to "sandbox"), which will probably need editing in the repro repo. There's also a bunch of AWS SSO and RBAC stuff, but I think that can largely be ignored (I yanked this from some existing code).

Also, the issue isn't present when "manually" creating k8s resources using the native TypeScript API.

Also, I realise the title is a bit poor, but I'm not sure

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[viveklak]

@omninonsense thanks for the great repro! Would you please consider upgrading your provider to 3.19.4 and double checking it is actually being used?

1. Remove the pulumi:disable-default-providers setting
2. Run an update with kubernetes dependency set to 3.19.4
3. Re-add the disable-default-providers flag

Also if this still occurs could you check the output of pulumi stack export and make sure the provider referenced is at 3.19.4? Thanks again!

[omninonsense]

Thanks for looking into this, @viveklak!

So, with (output from pulumi about)

Dependencies:
NAME                VERSION
@pulumi/aws         5.9.1
@pulumi/awsx        0.40.0
@pulumi/command     0.1.0
@pulumi/eks         0.41.0
@pulumi/kubernetes  3.19.4
@pulumi/pulumi      3.35.1
@pulumi/random      4.8.0
yaml                2.1.1
@types/node         14.18.21

I still get:

Diagnostics:
  pulumi:pulumi:Stack (infra-sandbox):
    error: TypeError: Cannot read properties of undefined (reading 'map')
        at /Users/nino/Documents/silicoai/infrastructure/pulumi/infra/node_modules/@pulumi/yaml/yaml.ts:2993:14
        at processTicksAndRejections (node:internal/process/task_queues:95:5)

The output of pulumi stack export (for an empty stack) is:

{
    "version": 3,
    "deployment": {
        "manifest": {
            "time": "2022-06-27T15:34:42.722246+01:00",
            "magic": "not sure if this is secret or not, so I removed it",
            "version": "v3.35.1"
        },
        "secrets_providers": {
            "type": "service",
            "state": {
                "url": "https://api.pulumi.com",
                "owner": "silico",
                "project": "infra",
                "stack": "sandbox"
            }
        }
    }
}

This fails during the plan/review phase (rather than when when applying). Also, this is not from the repro repo, but a real project, but the structure is largely the same.

[omninonsense]

Also, this seems to be happening regardless of whether I disabled or enabled the default Kubernetes provider.

[viveklak]

@omninonsense ah so this is failing with an empty/brand new stack?

[omninonsense]

@viveklak Yup, it doesn't seem to make a difference whether it's empty or if I add it to an existing stack

[viveklak]

Confirmed this is happening without disabling default kubernetes provider - you just need to use ConfigFile on the non-default provider. The problem seems to be that the invoke to kubernetes:yaml:decode returns an empty response. Interestingly I am not seeing an actual RPC call being made to the provider here but I need to do some more investigation here to confirm. Marked P1 and assigned myself.

[viveklak]

Investigating further - it is indeed the case that the invoke never actually makes it to the underlying provider. The issue in my repro was that if the underlying kubernetes provider has unknowns for its configuration, the provider is marked as unconfigured: https://github.com/pulumi/pulumi/blob/c9ba17d450f3717e006d7ce0be7736c4f0486447/sdk/go/common/resource/plugin/provider_plugin.go#L505 Later, any attempts to invoke on this provider results in an empty response: https://github.com/pulumi/pulumi/blob/c9ba17d450f3717e006d7ce0be7736c4f0486447/sdk/go/common/resource/plugin/provider_plugin.go#L1252-L1254

This is the reason why the call to kubernetes:yaml:decode is returning an empty response: https://github.com/pulumi/pulumi-kubernetes/blob/763ddcadd7843ea94e87239383da8d0a436b92bc/sdk/nodejs/yaml/yaml.ts#L2892

Granted an empty yaml body should be handled better here of course, but it seems unfortunate that we can't invoke the yaml decode function in previews (thus not be able to provide a preview of the resources to be created by loading the ConfigFile) if the provider has unknowns, since in this case its entirely feasible to perform the invoke even if the underlying kubernetes cluster is unavailable. At the very least, it would be good to allow the provider to handle this situation (preview/unconfigured provider) and not short circuit the request entirely within the engine.

Thoughts @Frassle?

[viveklak]

A hacky workaround for now would be to either:

1. Run the pulumi up with --skip-preview
2. Move the ConfigFile resource into an apply block wrapping the unknown configuration options of the provider, e.g.: if k8sprovider relies onkubeconfig which is unknown (output of cluster creation), move the ConfigFile to:

provider.kubeconfig.apply(_ => new k8s.yaml.ConfigFile("confFile"...))

[Frassle]

Granted an empty yaml body should be handled better here of course, but it seems unfortunate that we can't invoke the yaml decode function in previews (thus not be able to provide a preview of the resources to be created by loading the ConfigFile) if the provider has unknowns, since in this case its entirely feasible to perform the invoke even if the underlying kubernetes cluster is unavailable. At the very least, it would be good to allow the provider to handle this situation (preview/unconfigured provider) and not short circuit the request entirely within the engine.

So either we should only support Output based invokes, or we should only allow immediate values to provider construction. Because otherwise you end up in this state where we don't have a provider yet because of unknowns but you can start an invoke with a non-output response which can't indicate unknownness.

Short term we could error out the invoke if the provider was unknown? That's probably a better error than returning undef.

[mastoj]

I think I have the same issue in my small demo repo now. I create a aks cluster, than I want to use the config when creating the cluster to create a provider that I use to apply some kubernetes manifests, and it fails with:

TypeError: Cannot read property 'map' of undefined

If I use the default provider it seems to work.

@Frassle , I don't think allowing immediate values to create a provider is the way to go, especially with kubernetes. I can definitely see scenarios where you have your core infrastructure where you create your cluster AND also add common things to the cluster like istio, linkerd, traefik and things like that. In those scenarios I do think you want to create the cluster and then use the config you get from the cluster creation to create the provider to apply those things.

[viveklak]

Short term we could error out the invoke if the provider was unknown? That's probably a better error than returning undef.

@Frassle apologies I dropped the ball on following up here. Should we track an issue here on pu/pu to do the above?

[caboog]

This seems to happen in multiple ways. Similar issue, but the error I get in eks is this:

Exception: invoke of kubernetes:yaml:decode failed: Exception deserializing response!

[viveklak]

The plan here for the time being is to handle the empty response received in the various component resource implementations as a signal that no preview is available (due to an unconfigured provider). i.e. in such situations rich previews for resources under the config file etc. resources would not be possible.

[illinar]

Ran into the same issue with .NET SDK. It manifests somewhat differently by throwing NullReferenceException when creating ImmutableArray result. The issue is annoying since it is not possible to use previews when the stack relies on the newly created cluster credentials.

The workaround we have tried is to conditionally set provider arguments depending on whether the code runs in preview or not, like so:

new Kubernetes.Provider ("kube",
  Deployment.Instance.IsDryRun
    ? new Kubernetes.ProviderArgs ()
    : new Kubernetes.ProviderArgs { KubeConfig = cluster.KubeConfig })

What are the possible downsides of this approach?

For the record, the change that triggers this new behavior is 763ddcadd7843ea94e87239383da8d0a436b92bc

[viveklak]

Ran into the same issue with .NET SDK. It manifests somewhat differently by throwing NullReferenceException when creating ImmutableArray result. The issue is annoying since it is not possible to use previews when the stack relies on the newly created cluster credentials.

The workaround we have tried is to conditionally set provider arguments depending on whether the code runs in preview or not, like so:

new Kubernetes.Provider ("kube",
  Deployment.Instance.IsDryRun
    ? new Kubernetes.ProviderArgs ()
    : new Kubernetes.ProviderArgs { KubeConfig = cluster.KubeConfig })

What are the possible downsides of this approach?

For the record, the change that triggers this new behavior is 763ddca

Hi @illinar. This actually seems like a pretty reasonable workaround. As such the specific invokes in question don't actually leverage anything salient about the underlying kubernetes provider and its mostly used as a mechanism to offload yaml processing to a language agnostic helper in the provider through an RPC. The only issue with this is if you were to set the disable default providers option, the dry-run preview would fail.

[rhyek]

@illinar's suggestion works. For TypeScript:

import * as k8s from '@pulumi/kubernetes';
import * as pulumi from '@pulumi/pulumi';
import { cluster } from './cluster';

export const kubeProvider = new k8s.Provider('kube-provider', {
  kubeconfig: pulumi.runtime.isDryRun() ? undefined : cluster.kubeconfig,
});

and then:

export const dashboard = new k8s.yaml.ConfigFile(
  'dashboard',
  {
    file: 'https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml',
  },
  { provider: kubeProvider },
);

[rhyek]

The issue with the workaround is that after the initial deploy I get diffs during preview regarding the provider (though no changes are applied on actual update, of course):

[illinar]

Yes, we've realized this as well. Modifying provider state causes the diff between preview and deployed configurations 😞

[SharpEdgeMarshall]

This is breaking our production workloads please consider to revert PR that is causing this behaviour

[viveklak]

Apologies for the delays here. We have prioritized this issue and are actively working on it. We can't revert the PR as it is fundamental contract with the Pulumi engine to support other important use cases such as disabling default providers etc. We will instead follow the approach mentioned here: https://github.com/pulumi/pulumi-kubernetes/issues/2038#issuecomment-1197333626

Note if you are being blocked here, the recommended workaround for the moment is here: https://github.com/pulumi/pulumi-kubernetes/issues/2038#issuecomment-1169544108

[lblackstone]

I suspect https://github.com/pulumi/pulumi-kubernetes/issues/1987 may be related as well.

Edit: Confirmed. Here's a repro: https://github.com/phillipedwards/pulumi-helm-error