kind
kind copied to clipboard
kubernetes-sigs
WIP: Adds OCI image support
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: claudiubelu
To complete the pull request process, please assign bentheelder after the PR has been reviewed.
You can assign the PR to them by writing /assign @bentheelder in a comment when ready.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
What's the use case? https://github.com/kubernetes-sigs/kind/issues/2626
Also we're generally loathe to add dependencies (due to kind being embedded in tools / test suites / ...), though these seem fairly light.
What's the use case? #2626
Also we're generally loathe to add dependencies (due to kind being embedded in tools / test suites / ...), though these seem fairly light.
My main goal at this point is to be able to have official kube-proxy images for Windows (https://github.com/kubernetes/kubernetes/pull/109939). However, the current image building process for it won't work for it. In k/k, the images are first built with docker buildx, and then exported with docker save. However, that won't work with Windows images at all since they're not available / visible from docker itself (since you cannot have Windows images on Linux). Hence, my idea was to export the image directly from docker buildx, but it seems that docker buildx exports OCI images, which then causes the CIs that use kind to fail with error building node image: could not find image metadata because OCI images are not supported in kind.
However, this might not help me accomplish my goal at this point. I see in kubernetes/release that when the images are to be released, they are loaded with docker load, which would fail for Windows images because you cannot have Windows images on Linux. I was thinking of adding an option to k/k to allow the docker buildx build output to be registry, which would mean that the images would be uploaded directly to the given registry, however, I see in kubernetes/release that the image tars are also uploaded to GCS buckets, which means that the tars are still needed (and docker buildx build doesn't support multiple outputs, although there's a draft for that feature).
Hence, my idea was to export the image directly from docker buildx, but it seems that docker buildx exports OCI images, which then causes the CIs that use kind to fail with error building node image: could not find image metadata because OCI images are not supported in kind.
Ahhh.
We have also to consider Kubernetes' hack/local-up-cluster.sh, I think we've switched from docker to containerd there so we probably have OCI import support.
see in kubernetes/release that when the images are to be released, they are loaded with docker load, which would fail for Windows images because you cannot have Windows images on Linux.
This is probably a remnant for re-tagging / pushing them? We can probably rework those scripts to avoid actually using docker, https://github.com/google/go-containerregistry/tree/main/cmd/crane
I was thinking of adding an option to k/k to allow the docker buildx build output to be registry, which would mean that the images would be uploaded directly to the given registry, however, I see in kubernetes/release that the image tars are also uploaded to GCS buckets, which means that the tars are still needed (and docker buildx build doesn't support multiple outputs, although there's a draft for that feature).
I think we could take advantage of caching in buildx and build repeatedly with different output setting to produce the tars, or else we could fetch tars (crane pull) from the pushed images, but release may not like the ordering of having the images pushed first.
Also possible ctr image import doesnt support OCI yet ... https://github.com/kubernetes-sigs/kind/issues/2760#issuecomment-1125243694
Also possible
ctr image importdoesnt support OCI yet ... #2760 (comment)
containerd supports OCI images: https://github.com/containerd/containerd/blob/b9bffd1f38c7e85f433c22d0968cffb196ede000/images/archive/importer.go#L126 . It even goes further: if it's not an OCI image, it converts it to OCI.
Also possible
ctr image importdoesnt support OCI yet ... #2760 (comment)
This issue is a bit old, so my comment may not be relevant. This is a standard flow I use in my daily tinkering:
podman build [--squash-all] --tag myimage:0.0.1 .
podman save --format oci-archive --output ../myimage.0.0.1.tar localhost/myimage:0.0.1
KIND_EXPERIMENTAL_PROVIDER=podman kind load image-archive ../myimage.0.0.1.tar
I very rarely have a problem with that. The OCI format works perfectly well. Sometimes(very rarely) I need to resort to squashing the image to a single layer(--squash-all) to sort the problems with permissions(I work in rootless context most of times) and/or reduce the overall size of the image. However, I do work on Linux, so that's not quite the same context.
As a side note: I would love to see kind being able to consume the images directly from the docker/podman cache without the need for an interim step of saving the image as tarball or pushing it to a repo.. even if I can easily set such repo up on the same machine and keep pushing the images there. I find it equally (in)convenient to using the image dump. No real improvement in usability.
What would help here a little is to enrich kind with the option to pull the images straight from cache. I guess it would be best to do it explicitly by expressing the intent with an extra parameter in the CLI. I would like to avoid, is to guess the source of an image. We do have such prompt on podman already when using the short image names and multiple repos in the config and it may become a source of confusion very quickly. It would be better to use something like --from-build-cache, sniff the OCI complaint layers' store and work with that in order to pump the image to the cluster somehow.
There was some work done on this PR and I think it could be used as a starting point. Alternatively we could start from scratch and define a new set of requirements for such feature as I don't think it's only about OCI layout support. That support is already available I believe. It is more about taking out that extra interim translation (build -> cache -> archive -> cluster). It would be better still to get to the point where we have full transparency where the caching layer of the image builder is shared with the cluster i.e. it is mounted directly there so there is no transformation required. This could be pretty difficult to achieve though.
I could try to use my spare time and help in pushing it a bit further.. unless somebody tried that before and analysed such flow and there is no point in going there. Just to be clear - I love simplicity, so if it's not simple, I'd rather shoot these extra commands and settle for a reliable flow rather than implementing the monster which would be impossible to manage.
@BenTheElder : do you think it makes sense what I'm suggesting here?
consume the images directly from the docker/podman cache without the need for an interim step of saving the image as tarball or pushing it to a repo.
this is a very complex and brittle feature. at least in docker, e.g. the on-disk image storage is expressly not documented, before we even get into remote daemons. I'm pretty sure podman this is also the case (and podman also has nascent remote support now), not to mention builders like buildx.
image layers are only officially exposed in these tools by way of pushing to a registry, or by way of exporting a complete image or images.
Alternatively we could start from scratch and define a new set of requirements for such feature as I don't think it's only about OCI layout support. That support is already available I believe.
This PR is actually totally unrelated to loading images into a cluster and is about the images built by Kubernetes's build system, which are currently always docker tarballs. We need to handle them specially when building node images not quite in the same way as loading at runtime.
It would be better still to get to the point where we have full transparency where the caching layer of the image builder is shared with the cluster i.e. it is mounted directly there so there is no transformation required. This could be pretty difficult to achieve though.
yes, difficult, non-portable, and depending on internal details of other tools 😬
and somewhere we still have to create content ingestible by containerd, so we're still probably going to be converting to a tarball somewhere in the pipeline ...
@claudiubelu: PR needs rebase.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@claudiubelu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
| Test name | Commit | Details | Required | Rerun command |
|---|---|---|---|---|
| pull-kind-e2e-kubernetes-1-26 | 30494d17fb307f1dfb6115a316854697b8cd31b8 | link | true | /test pull-kind-e2e-kubernetes-1-26 |
| pull-kind-e2e-kubernetes-1-25 | 30494d17fb307f1dfb6115a316854697b8cd31b8 | link | true | /test pull-kind-e2e-kubernetes-1-25 |
| pull-kind-e2e-kubernetes-1-27 | 30494d17fb307f1dfb6115a316854697b8cd31b8 | link | true | /test pull-kind-e2e-kubernetes-1-27 |
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.