kubernetes-sigs
Feature 2160 add images subcommand
This PR addresses Feature 2160 - ability to include additional images in the kindest/node image and hopefully addresses the concerns in the discussion.
The PR:
- adds a subcommand to the build target
kind build add-image IMAGE [IMAGES...]in order to keep the command separate fromkind build node-image - pulls images locally before they are added into the resulting image to speed up future builds
- tars those images temporarily (similar to what kind load does)
- starts up the base image and use critl to copy/stream the images into it
- saves the image using docker save
Extras:
- follows the same patterns as node image when it comes to code layout and args
- has defaults for the base image used (kindest/node:latest) and the resulting image (kindest/custom-node:latest)
- can handle adding multiple images at the same time
An example of the command being used:
>> ./bin/kind build add-image --base-image=docker.io/kindest/node:v1.19.11 --image=myrepo/custom-node:latest alpine:3.7
Starting to add images to base image
Creating build container based on docker.io/kindest/node:v1.19.11
Building in kind-build-1623261338-1249758035
Saving images into tar file at /private/var/folders/81/myw19h2108xdx31prlv3qsd00000gn/T/images-tar466634204/images.tar
Importing images into build container kind-build-1623261338-1249758035
Saving new image myrepo/custom-node:latest
sha256:bc17c0f40f826574adb8aa886c29404b02c5e1fd89f2ff8d58f7cb55f59dce1e
Add image build completed.
>> cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: myrepo/custom-node:latest
EOF
Creating cluster "kind" ...
â Ensuring node image (myrepo/custom-node:latest) đŧ
â Preparing nodes đĻ
â Writing configuration đ
â Starting control-plane đšī¸
â Installing CNI đ
â Installing StorageClass đž
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community đ
>> docker exec -it kind-control-plane crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd v20210326-1e038dc5 6de166512aa22 54MB
docker.io/library/alpine 3.7 51af6a90e1cee 4.47MB
docker.io/rancher/local-path-provisioner v0.0.14 e422121c9c5f9 13.4MB
k8s.gcr.io/build-image/debian-base v2.1.0 c7c6c86897b63 21.1MB
k8s.gcr.io/coredns 1.7.0 bfe3a36ebd252 14MB
k8s.gcr.io/etcd 3.4.13-0 0369cf4303ffd 86.7MB
k8s.gcr.io/kube-apiserver v1.19.11 2fb26b7ebd23a 120MB
k8s.gcr.io/kube-controller-manager v1.19.11 1af1564e1890d 112MB
k8s.gcr.io/kube-proxy v1.19.11 d2c2a8b355f56 120MB
k8s.gcr.io/kube-scheduler v1.19.11 120d339f7d67a 47.7MB
k8s.gcr.io/pause 3.5 ed210e3e4a5ba 301kB
Please let me know if there is something you would like to see changed or fixed.
Welcome @curtbushko!
It looks like this is your first PR to kubernetes-sigs/kind đ. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.
You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.
You can also check if kubernetes-sigs/kind has its own contribution guidelines.
You may want to refer to our testing guide if you run into trouble with your tests not passing.
If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!
Thank you, and welcome to Kubernetes. :smiley:
Hi @curtbushko. Thanks for your PR.
I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
We may need to consider --arch support similar to kind build node-image. Unfortunately when doing that ...
pulls images locally before they are added into the resulting image to speed up future builds
That will not work once this supports multi-arch, perhaps best to head that off now. (see: https://github.com/kubernetes-sigs/kind/pull/2176)
We may need to consider
--archsupport similar tokind build node-image. Unfortunately when doing that ...pulls images locally before they are added into the resulting image to speed up future builds
That will not work once this supports multi-arch, perhaps best to head that off now. (see: #2176)
Is there a way I could use docker pull image:tag@sha and parse out the manifest through docker manifest inspect and still use the local docker cache?
What I am aiming for is being being able to build our product images locally and build a node image for all developers to use. That gets difficult if I have to pull images from a private registry while building.
Is there a way I could use docker pull image:tag@sha and parse out the manifest through docker manifest inspect and still use the local docker cache?
The problem I found is that if you need to ask docker to pull an image for a platform other than the host and then save it for loading you run into issues (e.g. try pulling the image for each platform you're building for and loading it, it will just save the host platform one, there's no way to save --platform), so the only solution I found was to pull for the platform directly to the node container we're going to commit.
I think there's more on this in the linked PR? But without a solution for that multi-arch support is untenable, and multi-arch support is very much needed now.
Also AFAIK there is no way to ask docker for layers locally (versus images by name), the only way docker exposes layers is through pushing to registries.
Thanks for the explanation and insight. I did go through your PR to see how you did it and I even took a dive into docker engine to see how it was doing save. It's too bad that they didn't add --platform to save even though they added it to other commands.
I found something very interesting! It looks like save does save the correct local image but it changes the manifest as it is writing it.
I pulled an odd platform version and built my custom image. I am on a mac.
>> docker pull --platform=s390x alpine:3.6
>> docker run alpine:3.6
WARNING: The requested image's platform (linux/s390x) does not match the detected host platform (linux/amd64) and no specific platform was requested
qemu-s390x: warning: 'msa5-base' requires 'kimd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'klmd-sha-512'.
>> ./bin/kind build add-image alpine:3.6
Starting to add images to base image
Creating build container based on docker.io/kindest/node:latest
Building in kind-build-1623283038-727782895
Saving images into tar file at /private/var/folders/81/myw19h2108xdx31prlv3qsd00000gn/T/images-tar588984380/images.tar
Importing images into build container kind-build-1623283038-727782895
Saving new image kindest/custom-node:latest
sha256:a362f1ce00e277e34acb9dc2423039f168cd76c6e48678725278d56bbee6de20
Add image build completed.
>> cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: myrepo/custom-node:latest
EOF
Creating cluster "kind" ...
....
Inspect the images locally and in the custom node. The rootfs layers match.
>> docker inspect alpine:3.6 |grep -A5 'RootFS'
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:f129ff7d977bdbcb6eb0ec9e2e24c6cbe68e60e46d9620e79e09e73ab2e545e5"
]
},
>> docker exec -it kind-control-plane crictl inspecti docker.io/library/alpine:3.6 |grep -A5 'rootfs'
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:f129ff7d977bdbcb6eb0ec9e2e24c6cbe68e60e46d9620e79e09e73ab2e545e5"
]
},
Even the creation dates are the same!
>> docker inspect alpine:3.6 |grep 'Created'
"Created": "2019-03-08T03:35:59.716135296Z",
>> docker exec -it kind-control-plane crictl inspecti docker.io/library/alpine:3.6 | grep 'created'
"created": "2019-03-08T03:35:59.716135296Z",
>> docker pull --platform=amd64 alpine:3.6 && docker inspect alpine:3.6 |grep 'Created'
3.6: Pulling from library/alpine
Digest: sha256:66790a2b79e1ea3e1dabac43990c54aca5d1ddf268d9a5a0285e4167c8b24475
Status: Image is up to date for alpine:3.6
docker.io/library/alpine:3.6
"Created": "2019-03-07T22:20:00.563496859Z",
Bonus: If I run an alpine pod in my custom node image, the logs give me:
> kubectl logs -f alpine
qemu-s390x: warning: 'msa5-base' requires 'kimd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'klmd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'kimd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'klmd-sha-512'
Just for fun, I pulled a mips version of redis, built it into my node and tried running it
kubeclt logs -f redis
qemu-mips64el: unable to find CPU model 'max'
Digging into this was quite fun!
@BenTheElder does it makes sense to duplicate the code for the docker commands pkg/build/nodeimage/internal/container/docker ?
@BenTheElder does it makes sense to duplicate the code for the docker commands
pkg/build/nodeimage/internal/container/docker?
In general I would say no but should it also go to a shared location?
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: curtbushko To complete the pull request process, please ask for approval from amwat after the PR has been reviewed.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@BenTheElder does it makes sense to duplicate the code for the docker commands pkg/build/nodeimage/internal/container/docker ?
eh, I prescribe to WET: Write Everything Thrice. Once or twice is not enough usage to design a good abstraction, DRY it out once you hit at least 3 times. but more importantly the code in any internal/ is not meant to be a publicly importable API (We did not put enough work into making it a good stable API, they are just implementation details of the good APIs), so it will still have to be private, and so far we have made the rule that pkg/cmd/ is doable using only it's own internal details or the public APIs, the CLI does not import any shared internals so we know the CLI commands are vetting the public APIs.
Any progress on this PR? This is extremly useful and is already used as a fork in multiple projekt for round about a year. No issues so far, so I don't see any reason to defer a merge. Clean-up tasks could be done in a follow-up PR!
This is a great feature. I just successfully tested it on my machine, great job! I'm using the fork currently to build custom node images containing a bunch of images I need to have in place in order to run a big application at my job. This saves me a lot of time and traffic. Great job, I really hope this makes it into the main repo soon! Cheers!
Looks like there are some comments to be addressed yet. Commits should also be squashed.
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: curtbushko Once this PR has been reviewed and has the lgtm label, please ask for approval from aojea. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: curtbushko Once this PR has been reviewed and has the lgtm label, please ask for approval from aojea. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@curtbushko: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
| Test name | Commit | Details | Required | Rerun command |
|---|---|---|---|---|
| pull-kind-verify | eab6fdb2025efac6881571e31ffb2d7df52f5af3 | link | true | /test pull-kind-verify |
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.