kubernetes
KEP-4639: update to most recent implementation state
- One-line PR description: bump https://github.com/kubernetes/enhancements/issues/4639 to beta
- Issue link: https://github.com/kubernetes/enhancements/issues/4639
- Other comments: @saschagrunert is on leave this week and is the primary author of this KEP, I am opening this in his stead to get it on PRR radar, but it may change once he returns.
I would defer to @saschagrunert on that one. From my perspective, since the kubelet is specifying a registry path, there's no reason from kube's perspective we'd limit it to a just images. We currently don't support artifacts because neither containerd nor CRI-O support them with this. That could be subject to change (and as I understand it there are conversations happening to change it), but from the kubelet's perspective it's just a string that the runtimes interpret
There are no implementations that support Artifacts, and discussions for how they could be supported were pushed off so there's no way for them to be implemented. OCI Artifacts are implicitly listed as a non-goal in the KEP. But because of the title, we've been seeing people complaining to OCI about why our spec is wrong.
Please remove references to OCI Artifacts from the KEP since those are explicitly not supported.
@sudo-bmitch this depends on the runtime, CRI-O for example supports OCI artifacts in a dedicated format. I'd say we leave this up to the container runtime implementation which types of images they support. Would that work?
I updated the PR, PTAL again @deads2k :pray:
@haircommander PTAL too
Please remove references to OCI Artifacts from the KEP since those are explicitly not supported.
@sudo-bmitch this depends on the runtime, CRI-O for example supports OCI artifacts in a dedicated format. I'd say we leave this up to the container runtime implementation which types of images they support. Would that work?
The CRI-O examples I've seen involve commands like:
# Push the artifact
oras push --config config.json:application/vnd.oci.image.config.v1+json ...
That is by definition not an OCI Artifact which depends on the config media type being something other than a known OCI media type. In this case, the selected media type defines the "artifact" as an OCI Image. The same issue goes for the example artifact quay.io/crio/artifact:v1 which is also an OCI Image according to the config media type. See https://github.com/opencontainers/image-spec/blob/main/artifacts-guidance.md for more details.
Are there CRI-O examples of mounting a Helm chart as a volume? I also have a few example artifacts you can try at docker.io/sudobmitch/demo:artifact and docker.io/sudobmitch/demo:artifact4.
Are there CRI-O examples of mounting a Helm chart as a volume? I also have a few example artifacts you can try at
docker.io/sudobmitch/demo:artifactanddocker.io/sudobmitch/demo:artifact4.
Unfortunately not, CRI-O does not support that yet. We plan for that in the future, though.
Unfortunately not, CRI-O does not support that yet. We plan for that in the future, though.
Maybe it's worth putting a blurb that says something along the lines of "The kubernetes community cannot guarantee what OCI objects are supported by which CRI implementations, as ultimately the parsing of an image name is done by the underlying CRI implementation. For instance, at the time of writing, neither containerd nor CRI-O support OCI artifacts, but that may change in the future" How do you feel about that @sudo-bmitch ?
Unfortunately not, CRI-O does not support that yet. We plan for that in the future, though.
Maybe it's worth putting a blurb that says something along the lines of "The kubernetes community cannot guarantee what OCI objects are supported by which CRI implementations, as ultimately the parsing of an image name is done by the underlying CRI implementation. For instance, at the time of writing, neither containerd nor CRI-O support OCI artifacts, but that may change in the future" How do you feel about that @sudo-bmitch ?
Back in the earlier discussions for the KEP, particularly when the idea of subject/referrers support was raised, I mentioned the complexity of the request:
- One image can have multiple referrers, and it's possible for multiple referrers to exist with the same artifact type.
- Each of those referrers is an artifact, and for any artifact mounted, it may have zero, one, or multiple layers.
- Each layer may have a different media type, and the content may be raw data/binary, or it may be a well known filesystem encapsulation format (like tar, zip, qcow).
The response to that was to push off those questions to a future KEP and limit this KEP to only support OCI images to avoid being slowed down. Otherwise, even to support artifacts without referrers, I believe you'd need enough data to define the filesystem structure, including the directory and filename, ownership, and permissions, for each layer in the artifact. I'm not sure how that can be done without more configuration options in the volume mount.
Removing the "OCI Artifact" from the title and description doesn't mean projects like CRI-O couldn't support it with their own syntax. But it's clarifying that the specification for how to support it has not been created yet for multiple projects to standardize against.
Given that there's so much confusion around whether this KEP supported OCI Artifacts, and end users are raising issues and contacting OCI about the alpha feature, I'm really concerned that moving this into beta with the implication that OCI Artifacts are supported would be harmful to the community. Additionally, removing it from the headline would make a future KEP, that adds support for OCI Artifacts, easier to distinguish from this KEP.
Unfortunately not, CRI-O does not support that yet. We plan for that in the future, though.
Maybe it's worth putting a blurb that says something along the lines of "The kubernetes community cannot guarantee what OCI objects are supported by which CRI implementations, as ultimately the parsing of an image name is done by the underlying CRI implementation. For instance, at the time of writing, neither containerd nor CRI-O support OCI artifacts, but that may change in the future" How do you feel about that @sudo-bmitch ?
Back in the earlier discussions for the KEP, particularly when the idea of subject/referrers support was raised, I mentioned the complexity of the request:
- One image can have multiple referrers, and it's possible for multiple referrers to exist with the same artifact type.
- Each of those referrers is an artifact, and for any artifact mounted, it may have zero, one, or multiple layers.
- Each layer may have a different media type, and the content may be raw data/binary, or it may be a well known filesystem encapsulation format (like tar, zip, qcow).
The response to that was to push off those questions to a future KEP and limit this KEP to only support OCI images to avoid being slowed down. Otherwise, even to support artifacts without referrers, I believe you'd need enough data to define the filesystem structure, including the directory and filename, ownership, and permissions, for each layer in the artifact. I'm not sure how that can be done without more configuration options in the volume mount.
Removing the "OCI Artifact" from the title and description doesn't mean projects like CRI-O couldn't support it with their own syntax. But it's clarifying that the specification for how to support it has not been created yet for multiple projects to standardize against.
Given that there's so much confusion around whether this KEP supported OCI Artifacts, and end users are raising issues and contacting OCI about the alpha feature, I'm really concerned that moving this into beta with the implication that OCI Artifacts are supported would be harmful to the community. Additionally, removing it from the headline would make a future KEP, that adds support for OCI Artifacts, easier to distinguish from this KEP.
Agree, when we add volume mounting of artifacts support it is likely we will need a very different api. Best to call that out now.
Agree, when we add volume mounting of artifacts support it is likely we will need a very different api. Best to call that out now.
We need a parallel effort across runtime communities (CRI-O, containerd, podman..) to agree on what it means to make an artifact available as a volume. It would have to be shared agreement, so that artifact mounting support is portable. Hopefully, that may fit under the current API but could also need changes based on the outcome of that effort.
Added another commit on top to remove the references to "OCI artifacts" to make it clearer that we will focus on OCI images within thie KEP.
I added a blurb for https://github.com/kubernetes/enhancements/pull/4897#pullrequestreview-2358345940 as well as specified the beta will be off by default until containerd and cri-o have a released RC including support
/close
according to https://github.com/kubernetes/enhancements/pull/4897#discussion_r1795950498
@haircommander: Closed this PR.
In response to this:
/close
according to https://github.com/kubernetes/enhancements/pull/4897#discussion_r1795950498
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
/reopen see comment https://github.com/kubernetes/enhancements/pull/4897#discussion_r1796139509
@mikebrow: Reopened this PR.
In response to this:
/reopen see comment https://github.com/kubernetes/enhancements/pull/4897#discussion_r1796139509
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
BTW, the key promotion isn't alpha to beta. The one to really be cautious about is off-by-default to on-by-default.
I would promote this to beta but off-by-default and then let a future minor release enable it by default.
I presume that the kubelet could try creating a local sandbox and test if the feature works. Not quite sure how (maybe the node doesn't have access to any registry that allows anonymous access).
Alternatively we could report a condition on the node for ImageVolumeSupport that starts Unknown and turns to true when the first Pod with an image becomes healthy. Doing that allows a distribution of Kubernetes to check the nodes as they come online, without making that a responsibility of core Kubernetes.
persisted data is problematic in upgrades, and beta has some cognotations for users , see my ocomment here and the thread in SIG architecture https://github.com/kubernetes/enhancements/pull/4897#discussion_r1796835468
@saschagrunert @haircommander Could we get the KEP update merged without the promotion? I think the changes are useful to add to the doc.
If we want to promote to beta, I think we would need an exception now.
@saschagrunert @haircommander Could we get the KEP update merged without the promotion? I think the changes are useful to add to the doc.
If we want to promote to beta, I think we would need an exception now.
I pushed an update to move the KEP back to alpha, I agree we should get at least the latest changes in.
/assign @mrunalp @dchen1107
We want to get the doc updates in but there was no exception so this will stay in alpha.
@mrunalp thank you for the review, I added your suggestions and squashed the commits together.
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Mark this PR as fresh with
/remove-lifecycle stale - Close this PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
It seems that the KEP need to be updated again.
In which direction? I assumed that the goal was to get it in and iterate on what we have.