pulumi-kubernetes

pulumi-kubernetes copied to clipboard

Problem description

The SDK types for Helm values are defined to allow Input values, but the actual implementations appear to require prompt/resolved values. This causes errors if the user passes unresolved Input values to the Chart.

I just hit this trying to install external dns with aws eks irsa and the dotnet sdk:

// external dns; https://github.com/bitnami/charts/tree/master/bitnami/external-dns
Logger.LogDebug("Installing external dns");
var externalDnsRole = new Role($"{prefix}-external-dns",
    new RoleArgs
    {
        AssumeRolePolicy = IamHelpers.AssumeRoleForServiceAccount(oidcArn, oidcUrl, "kube-system", "external-dns", awsProvider),
        InlinePolicies = { ["policy"] = ReadResource("ExternalDnsPolicy.json") }
    },
    new ComponentResourceOptions { Provider = awsProvider });

new Chart("external-dns",
    new ChartArgs
    {
        Namespace = "kube-system",
        FetchOptions = new ChartFetchArgs { Repo = "https://charts.bitnami.com/bitnami" },
        Chart = "external-dns",
        Version = config.ExternalDnsVersion,
        Values =
        {
            ["logLevel"] = "debug",
            ["policy"] = "sync",
            ["provider"] = "aws",
            ["registry"] = "txt",
            ["txtPrefix"] = "k8s.",
            ["serviceAccount"] = new InputMap<object>
            {
                ["annotations"] = new InputMap<object>
                {
                    ["eks.amazonaws.com/role-arn"] = externalDnsRole.Arn
                }
            }
        }
    },
    new ComponentResourceOptions { Provider = provider });

I currently have to use apply as follows:

externalDnsRole.Arn.Apply(roleArn =>
    new Chart("external-dns",
        new ChartArgs
        {
            Namespace = "kube-system",
            FetchOptions = new ChartFetchArgs { Repo = "https://charts.bitnami.com/bitnami" },
            Chart = "external-dns",
            Version = config.ExternalDnsVersion,
            Values =
            {
                ["logLevel"] = "debug",
                ["policy"] = "sync",
                ["provider"] = "aws",
                ["registry"] = "txt",
                ["txtPrefix"] = "k8s.",
                ["serviceAccount"] = new Dictionary<string, object>
                {
                    ["annotations"] = new Dictionary<string, object>
                    {
                        ["eks.amazonaws.com/role-arn"] = roleArn
                    }
                }
            }
        },
        new ComponentResourceOptions { Provider = provider }));

I took some observations and there seems to be two problems.

The use of InputMap seems to produce a JSON serialization error, possibly related to a breaking change in .NET6.

  kubernetes:helm.sh/v3:Chart (external-dns):
    error: Pulumi.ResourceException: The type 'Pulumi.InputMap`1[System.Object]' can only be serialized using async serialization methods. Path: $.Values.
       at Pulumi.Kubernetes.Helm.V3.Chart(string releaseName, Union<ChartArgs, LocalChartArgs> args, ComponentResourceOptions options)+(ValueTuple<Union<ChartArgsUnwrap, LocalChartArgsUnwrap>, ImmutableHashSet<Resource>> values) => { } [1]
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func)

The above problem has a workaround of using Dictionary (as seen in the above workaround).

Looking at how the output value is serialized, it looks like the serializer writes an empty object rather than a string, yielding a different error from the invoke RPC call:

error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field .metadata.annotations of type string")

The grpc log:

{
  "method": "/pulumirpc.ResourceProvider/Invoke",
  "request": {
    "tok": "kubernetes:helm:template",
    "args": {
      "jsonOpts": "{\"api_versions\":[],\"namespace\":\"kube-system\",\"values\":{\"logLevel\":\"debug\",\"serviceAccount\":{\"annotations\":{\"eks.amazonaws.com/role-arn\":{}}},\"provider\":\"aws\",\"txtPrefix\":\"k8s.\",\"registry\":\"txt\",\"policy\":\"sync\"},\"chart\":\"external-dns\",\"version\":\"7.3.2\",\"fetch_opts\":{\"repo\":\"https://charts.bitnami.com/bitnami\"},\"release_name\":\"external-dns\"}"
    }
  },
  "errors": [
    "rpc error: code = Unknown desc = failed to generate YAML for specified Helm chart: failed to create chart from template: YAML parse error on external-dns/templates/serviceaccount.yaml: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field .metadata.annotations of type string"
  ],
  "metadata": {
    "kind": "resource",
    "mode": "client",
    "name": "kubernetes"
  }
}

This is fixed in the upcoming Chart v4 resource. You can use output values and even use Pulumi Assets as values. Enjoy!