Failure when creating kind cluster for Kubernetes 1.21

[jpkrohling]

What happened: when adding a test matrix for operators I maintain (Jaeger and OpenTelemetry), I faced a problem in both projects when running the tests against Kubernetes 1.21. The same tests and commands work for 1.20 and 1.19, and I can also get a working kind cluster if I manually run kind create cluster --config kind-1.21.yaml, which makes me believe this is a problem in the interaction between kuttl and kind.

This is what I see as output to kuttl test when using 1.21:

$ kubectl-kuttl test --kind-config=kind-1.21.yaml
=== RUN   kuttl
    harness.go:457: starting setup
    harness.go:245: running tests with KIND.
    harness.go:174: temp folder created /tmp/kuttl642873189
    harness.go:138: Loading KIND config from kind-1.21.yaml
    harness.go:156: Starting KIND cluster
    harness.go:508: cleaning up
    harness.go:517: collecting cluster logs to tests/_build/artifacts/kind-logs-1623224039
    harness.go:563: removing temp folder: "/tmp/kuttl642873189"
    harness.go:569: tearing down kind cluster
    harness.go:589: fatal error getting client: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
--- FAIL: kuttl (120.85s)
FAIL

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

• Clone github.com/open-telemetry/opentelemetry-operator
• Run kubectl-kuttl test --kind-config=kind-1.21.yaml -- fails to create the cluster
• Run kind create cluster --config kind-1.21.yaml -- cluster gets created
• Run kubectl-kuttl test --kind-config=kind-1.20.yaml -- tests pass

Anything else we need to know?:

kind configuration files: kind-1.21.yaml kind-1.20.yaml kind-1.19.yaml

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
• KUTTL version (use kubectl kuttl version): KUTTL Version: version.Info{GitVersion:"0.10.0", GitCommit:"fc8c0f2", BuildDate:"2021-05-19T13:16:03Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Fedora 34
• Kernel (e.g. uname -a): 5.12.5-300.fc34.x86_64
• Install tools: ??
• Others:

[esatterwhite]

Confirmed, problem exists on 1.21, fine on 1.20

[kensipe]

hmm... boo... thank you so much.. we will look into this

[kensipe]

great feedback btw!

[esatterwhite]

@kensipe This appears to be specifically with bootstrapping the kind cluster. If I start the cluster manually, and set startKind: false it'll work just fine.

[esatterwhite]

I'm attaching the journal log from a recent attempt journal.log

[esatterwhite]

Actually updating to 0.15 seems to work past v1.26 now... Maybe this has been fixed