Build and release `kindest/node:VERSION` on kubernetes release

[chuckha]

Right now the node image that kind uses is published by hand by @BenTheElder or @munnerz. It would be good to get this into the release pipeline somehow so that when a new version of kubernetes is published we also get a new node image for kind to use. The tooling exists, it's just a matter of getting the image pushed to the correct place. See the tooling at https://github.com/kubernetes-sigs/kind/blob/master/hack/build/push-node.sh

For complete context, please see this slack thread.

There are a few approaches discussed in the thread:

1. Shove this into anago in a similar way that the conformance image works. This is easy but the downside is that anago is getting too big and we shouldn't be adding more stuff to anago. This may become a slippery slope and we don't want anago to grow.

2. periodically check if the release tags have changed. If they have changed, do a build, otherwise don't do anything.

3. @BenTheElder has been looking at extending prow to trigger off GCS/GCR pubsub so we can kick off a build after a normal release is finished.

We would like to move this off dockerhub before 1.0, but there isn't really a great place to put it right now. Ideally we could have a CNCF sponsored gcr.io bucket so that google isn't responsible for the storage.

[neolit123]

i guess, i'm only -1 only on the anago approach.

We would like to move this off dockerhub before 1.0, but there isn't really a great place to put it right now. Ideally we could have a CNCF sponsored gcr.io bucket so that google isn't responsible for the storage.

this overlaps with the work by the k8s-infra-team, so if they manage to do the above soon the kind image can be on CNCF ground, otherwise dockerhub seems like a viable option for 1.0.

[BenTheElder]

Ditto w/ @neolit123 on anago per our previous discussion, I don't think Kubernetes the core project should be in the business of releasing SIG subprojects just yet, we can ensure we publish images for each release by other means.

For now if anyone really wants them today, they can check out a k8s release tag and build an "unnoficial image" with the same tools we use.

I'm also ambivalent on dockerhub, there haven't been major downsides so far but I'd be happy to move it to whatever hosting k8s-infra-team settles on going forward. The current dockerhub was indeed a stopgap to have joint access with @munnerz but has worked fine.

[chuckha]

cross posting a few communication links

email to steering committee about this https://groups.google.com/a/kubernetes.io/forum/#!topic/steering/ASZGGcvJQts

CNCF slack discussion about this https://cloud-native.slack.com/archives/C08PSKWPN/p1546470742035300

k8s-infra slack message about this https://kubernetes.slack.com/archives/CCK68P2Q2/p1546528406005400

tracking issue https://github.com/kubernetes/k8s.io/issues/158

[BenTheElder]

@munnerz and I have been looking into this more. We think we can set up jobs based on kubernetes/kubernetes git tags to publish new images.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[BenTheElder]

/remove-lifecycle stale

[BenTheElder]

we're at least not too far behind doing it manually, the current pattern is to publish a suite of images to go with each kind release, and avoid pushing much otherwise so users stick to stable image + kind combos

obviously this is not what we wanted here, but it's doing okay as a stopgap

for 0.4.0 releasing ~now we have:

• kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478
• kindest/node:v1.14.3@sha256:583166c121482848cd6509fbac525dd62d503c52a84ff45c338ee7e8b5cfe114
• kindest/node:v1.13.7@sha256:f3f1cfc2318d1eb88d91253a9c5fa45f6e9121b6b1e65aea6c7ef59f1549aaaf
• kindest/node:v1.12.9@sha256:bcb79eb3cd6550c1ba9584ce57c832dcd6e442913678d2785307a7ad9addc029
• kindest/node:v1.11.10@sha256:176845d919899daef63d0dbd1cf62f79902c38b8d2a86e5fa041e491ab795d33

[aojea]

Im publishing an unofficial image built using kind master and kubernetes master nightly

The image is in dockerhub aojea/kindnode:latest https://cloud.docker.com/repository/docker/aojea/kindnode

This is the repo that kicks off the job in circle ci https://github.com/aojea/kind-images

Example of a successful job https://circleci.com/gh/aojea/kind-images/19

[BenTheElder]

FWIW the v1.17.0 kind node image released within ~minutes of the kubernetes release (not automated yet, just bentheelder-bot watching the release ;-))

[drewwells]

v1.15.7 is the latest vailable, v1.15.10 is not available. I don't see a clear way to even help make this image available.

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/quick-start/#building-images

we currently more or less only publish "official" kind images with each kind release. this is because we still need to make breaking changes sometimes (right now related to #148).

you can build your own images, which ideally are built and consumed by the same kind version currently.

I'm going to overhaul building images in the future. #148 is the current priority.

[mflendrich]

I'd like to request availability of kindest/node images for pre-release versions of Kubernetes (alpha, beta).

We intend to utilize kind for regression testing against multiple versions of Kubernetes. Being able to include the upcoming Kubernetes version (which is available as an alpha/beta build) in our tests would shorten our feedback loop by weeks/months.

cc @shaneutt

[BenTheElder]

Currently you'll need to build your own, we have documentation for this in user guide. In the near future we will offer builds that don't require building Kubernetes from source but will result in slightly larger images.

[BenTheElder]

I expect to propose a concrete plan for this, #381, #166, and #1895 in the next month.

[BenTheElder]

We intend to utilize kind for regression testing against multiple versions of Kubernetes. Being able to include the upcoming Kubernetes version (which is available as an alpha/beta build) in our tests would shorten our feedback loop by weeks/months.

For this one specifically there's the additional constraint that using kind with versions of kubernetes that released after kind released may not work.

While Kubernetes shies away from making breaking end-user changes without a clear migration path over multiple releases, Kubernetes / kubeadm do have [Action Required] changes in a single release sometimes (e.g. https://github.com/kubernetes/kubeadm/issues/2376 landing in Kubernetes v1.21.0).

Because of this we must test Kubernetes at HEAD with kind from HEAD containing mitigations for these pre-release changes. You could use kind from HEAD yourself but currently it may also have breaking changes between releases and not yet have a detailed migration path etc.

This is a trick problem that may improve (e.g. perhaps kind will be more complete and release smaller bugfixes more often as we won't be in the middle of larger changes) but currently it somewhat limits the usefulness of us pre-building these versions. https://github.com/kubernetes-sigs/kind/issues/381 will at the very least make it cheaper to try without us having published one anyhow.

[mkarg]

I wonder if anything has improved over the past year? 🤔

[BenTheElder]

You can now build node images without building [kubernetes] from source, starting in kind v0.24.0, recommended for kubernetes 1.31+ (though it does work for older releases).

There's no guarantee that they'll work without staying atop of the latest kind changes though, because that's impossible to guarantee, for example the kubeadm config API is still in beta.

[mkarg]

That is amazing, but actually my status question was targeting the original topic of this thread:

Right now the node image that kind uses is published by hand by @BenTheElder or @munnerz. It would be good to get this into the release pipeline somehow so that when a new version of kubernetes is published we also get a new node image for kind to use. The tooling exists, it's just a matter of getting the image pushed to the correct place.

[BenTheElder]

That is amazing, but actually my status question was targeting the original topic of this thread:

This is one of the things discussed above related to it, we don't want another heavy source build in the CI.

But the other issues related to tagging and compatibility are not resolved.

However this makes it relatively cheap to build your own images, and is one of the pre-requisites for this issue.