RuntimeError when trying to install Calico via YAML

[pasmon]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

Trying to install Calico CNI with Pulumi via YAML files fails with: RuntimeError: Set changed size during iteration

Steps to reproduce

1. Create a Python Pulumi program, trying to install Calico as instructed here: https://projectcalico.docs.tigera.io/getting-started/kubernetes/k3s/multi-node-install

"""A Kubernetes Python Pulumi program"""
import pulumi
from pulumi import ResourceOptions, Output

from pulumi_kubernetes.yaml import ConfigFile

calico_operator = ConfigFile(
    "calico-operator",
    file="https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml",
)

calico = ConfigFile(
    "calico",
    file="https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml",
    opts=pulumi.ResourceOptions(depends_on=calico_operator)
)

2. Run pulumi preview
3. Watch the world burn, no really just Pulumi crashing :(

Expected: Calico CNI is installed and Pulumi continues to install the rest of the stack.

Actual: Pulumi preview fails pulumi-preview.txt

Pulumi version: v3.24.1 Pulumi Python package: 3.24.1 pulumi-kubernetes version: 3.15.1

[lukehoban]

Importantly - here's the last part of the stack trace:

      File "/home/pasmon/git/homelab-python/__pypackages__/3.8/lib/pulumi/output.py", line 123, in get_value
        val = await self._future
      File "/home/pasmon/git/homelab-python/__pypackages__/3.8/lib/pulumi/runtime/resource.py", line 514, in do_register
        resolver = await prepare_resource(res, ty, custom, remote, props, opts, typ)
      File "/home/pasmon/git/homelab-python/__pypackages__/3.8/lib/pulumi/runtime/resource.py", line 109, in prepare_resource
        explicit_urn_dependencies = await _resolve_depends_on_urns(
      File "/home/pasmon/git/homelab-python/__pypackages__/3.8/lib/pulumi/runtime/resource.py", line 851, in _resolve_depends_on_urns
        return await _expand_dependencies(all_deps, from_resource)
      File "/home/pasmon/git/homelab-python/__pypackages__/3.8/lib/pulumi/runtime/resource.py", line 822, in _expand_dependencies
        await _add_dependency(urns, d, from_resource)
      File "/home/pasmon/git/homelab-python/__pypackages__/3.8/lib/pulumi/runtime/resource.py", line 801, in _add_dependency
        for child in res._childResources:
    RuntimeError: Set changed size during iteration

[betamike]

I hit this bug as well with the same stack trace. My code has a ConfigGroup resource that was put in the depends_on block of another resource. As a test, I changed the ConfigGroup to ConfigFile and also experienced the bug.

Rough code (within a custom ComponentResource)

        cas = ConfigGroup(
            "gcasissuer-config",
            files=["google-cas-issuer-v0.5.3.yaml"],
            opts=ResourceOptions(parent=self),
        )

        cas_issuer = CustomResource(
             "google-cas-issuer",
             # [omitted]
             opts=ResourceOptions(parent=self, depends_on=[cas]),
        )

File "[omitted]/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 514, in do_register
    resolver = await prepare_resource(res, ty, custom, remote, props, opts, typ)
File "[omitted]/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 109, in prepare_resource
    explicit_urn_dependencies = await _resolve_depends_on_urns(
File "[omitted]/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 853, in _resolve_depends_on_urns
    return await _expand_dependencies(all_deps, from_resource)
File "[omitted]/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 824, in _expand_dependencies
    await _add_dependency(urns, d, from_resource)
File "[omitted]/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 804, in _add_dependency
    await _add_dependency(deps, child, from_resource)
File "[omitted]venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 803, in _add_dependency
    for child in res._childResources:
RuntimeError: Set changed size during iteration

Python Library Versions: pulumi 3.26.1 pulumi-gcp 6.16.0 pulumi-kubernetes 3.17.0

[AaronFriel]

@pasmon and @betamike thanks for your patience, getting back to this, it looks like there's an issue with our Kubernetes provider and using "depends_on" with resources that have children. You're likely to encounter this with ConfigFile, ConfigGroup, and Helm Charts.

A workaround is to use this in the resource options:

calico = ConfigFile(
    "calico",
    file="https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml",
    opts=pulumi.ResourceOptions(
                 # ⬇️ parent resource 
        depends_on=calico_operator.resources.apply(lambda x: list(x.values())),
                                 # ⬆️ wait on all child resources 
    )
)

I also noticed that the Calico/Tigera operator manifest sets a field, "status", we don't allow in CRDs, so I've copied below a full working program with both workarounds:

"""A Kubernetes Python Pulumi program"""
from importlib.resources import Resource
from typing import Any, Dict, List
import pulumi
from pulumi import ResourceOptions, Output

from pulumi_kubernetes.yaml import ConfigFile

def _fix_status(obj: Any, opts: pulumi.ResourceOptions):
    if obj.get("kind") == "CustomResourceDefinition":
        if "status" in obj:
            # Remove status field from CRD, a field we do not permit setting on creating CRDs.
            del obj["status"]

calico_operator = ConfigFile(
    "calico-operator",
    file="https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml",
    transformations=[_fix_status]
)


calico = ConfigFile(
    "calico",
    file="https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml",
    opts=pulumi.ResourceOptions(
        depends_on=calico_operator.resources.apply(lambda x: list(x.values())),
    )
)

[viveklak]

@pasmon and @betamike thanks for your patience, getting back to this, it looks like there's an issue with our Kubernetes provider and using "depends_on" with resources that have children. You're likely to encounter this with ConfigFile, ConfigGroup, and Helm Charts/Releases.

Helm release is not a component resource so shouldn't suffer from this. We rolled essentially your workaround into the chart resource implementation as part of https://github.com/pulumi/pulumi-kubernetes/issues/861 but I assume we need to do the same for configgroup and configfile. I will transfer this to the pulumi-kubernetes provider repo.

[pbarker]

Any update here? Hitting this today trying to install knative and istio from ConfigFile resources

[AaronFriel]

@pbarker the depends_on workaround above should help with this, can you let me know if it does?

[pasmon]

I bumped into this issue also with Kyverno ConfigFile, defining the Kyverno policies depending on the actual Kyverno installation. In this case, the depends_on workaround works for me. Thanks @AaronFriel !