enhancements
enhancements copied to clipboard
kubernetes
Add Image Decryption KEP
KEP for adding support for encrypted images in Kubernetes.
Signed-off-by: Harshal Patil [email protected]
Hi @harche. Thanks for your PR.
I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
:memo: Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.
It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.
- If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
- If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
- If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
- Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.
/ok-to-test
Also, @harche whenever you get a change, you'll need to sign the CLA before this can be merged (if it ends up being merged). thanks :)
@mattjmcnaughton Thanks. I plan to sign the CLA as soon as I can. Getting ready to travel for kubecon :)
I can definitely consider adding the section "Alternatives Considered"
On Fri, May 17, 2019 at 09:48:49AM -0700, harche wrote:
@mattjmcnaughton Thanks. I plan to sign the CLA as soon as I can. Getting ready to travel for kubecon :)
I can definitely consider adding the section "Alternatives Considered"
-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kubernetes/enhancements/pull/1066#issuecomment-493521827
Oh very exciting - enjoy Kubecon :)
For the very small part that impacts sig-node (CRI), lgtm. Not at a comment on the larger design.
can you add a section specific to when/how we would expect the image manager to pull images based on image pull policy and decryption keys? for example, if pull policy was "Never" but decrypt keys were provided, we would still call pull image (which the CRI may actually still pull which was not necessarily desired). This leads me to wonder if we need to just have authorization call separate from pull image to distinguish the two states.
That's a really good point. We can do up a matrix of these interactions which would also be very helpful documentation-wise as well.
We did have an initial implementation where we passed the ImageDecryptSecrets with CreateContainer which would call CheckAuthorization, but decided against that since ImageDecryptSecrets would naturally be associated with image related APIs than container related APIs in the CRI.
@derekwaynecarr @Random-Liu I have removed the constrains around imagePullPolicy. Now we are performing the Image Authorization irrespective of the imagePullPolicy.
I have update the KEP, https://github.com/kubernetes/enhancements/blob/76c8282441542361caf39e1e1addb9e659c8f4e5/keps/sig-node/20190517-image-decryption.md#Relationship-with-imagePullPolicy and the corresponding reference implementation, https://github.com/kubernetes/kubernetes/pull/78975
For easy reference: copying meeting notes from sig-node meeting Aug 13 2019
Next Step:
- Need approval from @derekwaynecarr and @dchen1107
- @Random-Liu will be the reviewer from sig-node.
@mtrmac @Random-Liu Thanks for your feedback, I have updated the document accordingly.
@Random-Liu @yujuhong @derekwaynecarr @smarterclayton I have updated the doc based on the review comments so far. I will be happy to answer if you have more questions.
Hi @smarterclayton ! Do let us know if we can help with assisting in the api-review process. Since we are tagged with 1.17, i believe the deadline to merge this in is 14 October. Would be good to hear any comments as you have them so that we can make the changes (if necessary) timely. Thanks!
@smarterclayton Does OpenShift support isolation of secrets between user for multi-tenancy purposes?
@smarterclayton Thanks for the comments! Understand your concerns here. Replies below.
The KEP also needs to discuss how container runtimes that opt not to implement this extension would signal that, how errors propagated to the Kubelet would be reflected all the way back to the user interface, and how a conformant Kubernetes cluster would react if this capability is not present.
:+1: Agreed on this. We'll add a section on this.
I'd like to see the KEP deal with why this must be a field on pods, vs implemented by container runtimes and out of scope for Kubernetes. Specifically, discussion of the threat model that this KEP is working against where secrets (visible to any administrator of the cluster) are being used to decrypt images, vs node level secrets that the container runtime uses to decrypt the contents.
You are exactly right that there are 2 different security models here. One that is node based, and one that is user based. Node based implementation does not require kubernetes integration (in most cases) - and this is something that we've implemented today in containerd and cri-o.
The usecase for the KEP is to provide multitenancy. Based on our discussions with the runtime community, this was the most appropriate course of action.
:+1: We will add this to the KEP.
I'd also like to see some ecosystem adoption (OCI + at least one container runtime) before we move this forward to Kubernetes. I don't think 1.17 is an appropriate milestone for an in progress work (although I welcome the KEP and PRS or feature branches that demonstrate this behavior).
We already have containerd adoption here. And are working with @rhatdan and team to have it up soon in the cri-o stack. I'm not opposed to delaying this to a later release. However, I do have a question about this.
Since the definition of this KEP does not actually have any association with OCI - i.e. how layer +zstd compression just works in runtimes now without any changes in Kubernetes. Is it reasonable that the measure for ecosystem adoption be "at least 2 inter-operable container runtimes"?
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
/remove-label api-review
While we ensure that we are meeting the bar here for api review ("The goal of the proposed change has agreement from the appropriate sub-project owners and/or SIG leads") while it evolves.
Could you illustrate an example flow of creation of a pod with this addons would look like?
On Sun, Feb 23, 2020, 7:47 AM Tim Bannister [email protected] wrote:
@sftim commented on this pull request.
In keps/sig-node/20190517-image-decryption.md https://github.com/kubernetes/enhancements/pull/1066#discussion_r383001187 :
+As seen in the diagram https://imgur.com/zQaAPp5, when the kubelet wants to create a new pod which has encrypted images it has to first retrieve the referenced
ImageDecryptSecretswhich hold the decryption keys (They can be referenced directly in the pod or deployment yaml or in the pod’s service account).
+Kubelet sends the request to pull the image along with the decryption keys to CRI which then forwards it to Containerd/CRI-O. Containerd/CRI-O looks up for the image in the corresponding snapshotter, if the image doesn’t exist then it's downloaded and decrypted using the decryption keys that were passed via CRI.
+
Image Authorizationis the process that only attempts to unwrap the keys in the image manifest of the image. In simple terms, it means everytime you want to use an encrypted image, you will have to prove that you have the necessary keys to decrypt it even if the actual image is present in the decrypted form in the snapshotter. This prevents an attack where a user, without having decryption keys, might get access to encrypted image content if their pod gets scheduled on a worker node where that particular encrypted image was already pulled and decrypted by earlier request.
+In case Containerd/CRI-O finds the image in the snapshotter cache which was already pulled and decrypted by earlier request, it will not perform the
Image Authorizationas a part of the call to pull the images. Kubelet makes a subsequent call toCreateContainerand passes the decryption keys with it, which a runtime like containerd/CRI-O can use to perform the image authorization. The advantage of performing image authorization duringCreateContaineris that it allows the image authorization to take place irrespective of theimagePullPolicyof the pod.
+## Alternatives Considered
+In order to decrypt an image containerd needs to have access to the corresponding private key. By the nature of it, a private key is a very sensitive piece of data. If it's lost, image confidentiality is compromised. It's a kind of a
secretthat's securing the data in an encrypted image. Kubernetes already has an infrastructure to handle secrets. This was the motivation to extend existing secrets to handle private keys required to decrypt an image.
+Containerd (or CRI-O) is the component that actually pulls the image, and hence does the decryption, on the worker node. So alternatively, if containerd manages to fetch keys on its own then we do not need Kubernetes to provision them. We did a POC around this idea where containerd talks to a Key Management Service (KMS) provider to fetch appropriate keys before decrypting the image. While this solution works as intended, the user needs to set up and maintain KMS.
+Using existing secret management in Kubernetes to provide decryption keys simplifies user flow, although using containerd to fetch keys also has it's use cases (mainly where the k8s master is not well trusted).
I was thinking that the add-on could expose the symmetric key either as a Secret or as a Service that authenticates callers and then talks gRPC.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/enhancements/pull/1066?email_source=notifications&email_token=AAXLDBVTCQDIA346SZCQWOTREJV7JA5CNFSM4HNSIV52YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCWSFUZA#discussion_r383001187, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXLDBVIPZJ72DTVOHFG2OLREJV7JANCNFSM4HNSIV5Q .
/milestone clear
(removing this issue from v1.17 milestone as the milestone is complete)
Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
:memo: Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.
It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.
- If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
- If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
- If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
- Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.