image-spec
image-spec copied to clipboard
opencontainers
Proposal to add Encrypted Layer Mediatype
This is a first draft of modifications to the OCI spec on Encrypted Container Images. Work around this has been on-going for quite a while (since the start of the issue). The current design is based on the discussions we've had in the issue and with the runtime implementation in containerd.
Abstract
We would like to propose a new media type for encrypted layers of a container image. This addition would facilitate the ecosystem for encrypted container images. This allows users with stricter trust requirements to be able ensure end-to-end encryption from build to runtime. In addition, it allows users to use a centralized managed repository (i.e. Docker Hub) without any risk of their images being compromised.
Issue Details
Based on the discussion of https://github.com/opencontainers/image-spec/issues/747
Implementations
The specification and libraries to perform encryption/decryption
The implementation to perform encryption and decryption of this spec is in:
- https://github.com/containers/ocicrypt
func EncryptLayer(ec *config.EncryptConfig, encOrPlainLayerReader io.Reader, desc ocispec.Descriptor) (io.Reader, EncryptLayerFinalizer, error)func DecryptLayer(dc *config.DecryptConfig, encLayerReader io.Reader, desc ocispec.Descriptor, unwrapOnly bool) (io.Reader, digest.Digest, error)
Container Runtimes
Implemented and upstreamed for containerd stack
- containerd (1.3+)
- https://github.com/containerd/imgcrypt
Implemented and upstreamed for RedHat stack
Registry
Current docker distribution supports: https://github.com/docker/distribution. Tested with v2.7.1.
Build toolchain
The plan for this is to have docker CLI, skopeo and buildah. In the meantime, we also support encryption of images in containerd via imgcrypt ctr client.
Current implementations for RedHat stack
Current implementations for containerd stack
- ctr-enc
- To integrate into docker CLI, we are currently waiting on https://github.com/moby/moby/issues/38043. Tracking with https://github.com/moby/buildkit/issues/714
Kubernetes Integration
It is important to note that kubernetes integration is not required to use encrypted container images. i.e. in cri-o you are able to specify a directory containing keys.
A way to do this is via an operator k8s-enc-image-operator
Signed-off-by: Brandon Lum [email protected]
@vbatts As per our conversation a couple days ago, Based on the RFC6838 and RFC6839.
A specification like nondistributable layers would work:
application/vnd.oci.image.layer.enc.tar.v1application/vnd.oci.image.layer.enc.tar.gzip.v1
The other way to do this is to have one encrypted layer mediatype application/vnd.oci.image.layer.enc.v1 and specify the original underlying mediatype in an annotation. But i'm unsure if this may make handling on the runtimes a bit more difficult.
cc: @estesp @dmcgowan @crosbymichael @mrunalp @mtrmac @vrothberg
I don't think reproducing what was done for nondistributable layers is a good idea. Runtimes end up having to special case these since they are hard to interpret. For example vnd.oci.image.layer. may indicate that an object should be treated as a layer, but then .tar.v1 indicates it is a tar. What is sandwiched in between those is some important metadata. In the nondistributable case it is just a hint of how an artifact is distributed, in the encrypted case, it is a requirement for how the tar must be processed. I think we really need to format the content types in a way that indicates how they should be processed, ensuring that modifications on those types are wrapping. Personally I like using + to indicate encoding (whether that is encryption or formatting). I think we should discuss at an upcoming OCI weekly meeting about these types and their relation to RFC6838.
Updated based on discussions from OCI weekly dev discussion to move from +enc to +encrypted. We agreed on using suffixes on mediatypes and having these defined on top of artifacts.
@mikebrow Responding to https://github.com/opencontainers/image-spec/issues/791#issuecomment-533394562 in this thread instead since it is more specific to encryption details.
The main integration points for Encrypted Container Images are:
- The specification and libraries to perform encryption/decryption
- Container runtimes
- The build toolchain
- Registry
- Kubernetes Integration
Do let me know if there's something specific you are looking for and I can provide a direct link.
The specification and libraries to perform encryption/decryption
The definition of encryption indication and how encryption metadata is defined in:
- https://github.com/opencontainers/image-spec/pull/775
The implementation to perform encryption and decryption of this spec is in:
- https://github.com/containers/ocicrypt
func EncryptLayer(ec *config.EncryptConfig, encOrPlainLayerReader io.Reader, desc ocispec.Descriptor) (io.Reader, EncryptLayerFinalizer, error)func DecryptLayer(dc *config.DecryptConfig, encLayerReader io.Reader, desc ocispec.Descriptor, unwrapOnly bool) (io.Reader, digest.Digest, error)
Container Runtimes
Implemented and upstreamed for containerd stack
- containerd (1.3+)
- https://github.com/containerd/imgcrypt
Implemented and upstreamed for RedHat stack
Registry
Current docker distribution supports: https://github.com/docker/distribution. Tested with v2.7.1.
Build toolchain
The plan for this is to have docker CLI, skopeo and buildah. In the meantime, we also support encryption of images in containerd via imgcrypt ctr client.
Current implementations for RedHat stack
Current implementations for containerd stack
- ctr-enc
- To integrate into docker CLI, we are currently waiting on https://github.com/moby/moby/issues/38043. Tracking with https://github.com/moby/buildkit/issues/714
Kubernetes Integration
It is important to note that kubernetes integration is not required to use encrypted container images. i.e. in cri-o you are able to specify a directory containing keys.
cc: @stefanberger, @harche
@lumjjb ok, finally discussing this in the weekly call. Since this encrypted layer discussion has begun, the https://github.com/opencontainers/artifacts has come along. To a large degree, this is another mimetype/mediaType that would be documented there, and allowed as an extension to the image-spec.
There is a lot of good work here in the PR, that should transfer to something like a README for these mime-types in the artifacts repo. Does this make sense?
@lumjjb ok, finally discussing this in the weekly call.
Awesome, sounds good.. I will come by the next OCI call on Wednesday.
There is a lot of good work here in the PR, that should transfer to something like a README for these mime-types in the artifacts repo. Does this make sense?
Yup! I will create an encryption.md PR to the artifacts repo, and I suppose it would be an edit pointing to that file from https://github.com/opencontainers/artifacts/pull/13/ ?
@AkihiroSuda it seems to be stalled on https://github.com/opencontainers/artifacts/pull/15
As a quick update on this, I've updated the list of links in the comments above.
The encryption support is now part of upstreamed containerd, cri-o, buildah, docker registry and skopeo tools.
@vbatts I'm a bit late to the party. But this has recently come into scope for me.
It isn't clear to me from the various pieces of documentation strewn about whether the integrity protection for a given layer also provides integrity protection for all the layers beneath it.
For example, you have an image with multiple layers:
- Layer 1
- Layer 2
- Layer 3 (encrypted)
Does the integrity protection of layer 3 also include measurements of layers 2 and 1?
If not, it would be possible to modify one of the intermediate layers in order to gain backdoor access to the plaintext.
Hi @npmccallum The encryption is only targeting confidentiality, although it has some side effects of integrity on a per layer basis. This was designed to be used in conjunction with Docker Content Trust / Notary / RH Simple signing, they should be able to work hand in hand.
@lumjjb If you don't have integrity of the whole, you don't have confidentiality of the layer.
If i understand your statement correctly, I believe those signing technologies ensure the integrity of the image manifest, and the integrity of the layers are ensured as being part of the signed manifest hash. This is done as a side effect of layers being content hashes.
@lumjjb Okay. So this is working off the presumption that you have integrity protection for the whole manifest which contains measurements of all the layers. Therefore, there is integrity protection for the whole and confidentiality protection for the parts.
During deployment, do you ensure that the manifest is validly signed and all the layers match their measurements before you decrypt? Or is it possible to decrypt without validating a signature of the manifest and validating the measurement of the layers?
During deployment, do you ensure that the manifest is validly signed and all the layers match their measurements before you decrypt? Or is it possible to decrypt without validating a signature of the manifest and validating the measurement of the layers?
Yes, this is usually done by the pull/admission phase of the image, so integrity is measured before decryption is done during the unpacking phase.
If not, it would be possible to modify one of the intermediate layers in order to gain backdoor access to the plaintext.
How would that be possible? We are encrypting the tar'ed and gzip'ed layers using AES_256_CTR_HMAC_SHA256 and a random IV. The symmetric encryption parameters, including the key and IV, are contained in a JSON document which is encrypted for each recipient using an RSA key for encryption with JWK, PKCS7, PKCS11, and gpg.
@lumjjb Do you refuse to decrypt if the manifest is unsigned?
That is the choice of the runtime policy, the encryption technology is agnostic to this, depending on the setup
If not, it would be possible to modify one of the intermediate layers in order to gain backdoor access to the plaintext.
How would that be possible? We are encrypting the tar'ed and gzip'ed layers using
AES_256_CTR_HMAC_SHA256and a random IV. The symmetric encryption parameters, including the key and IV, are contained in a JSON document which is encrypted for each recipient using an RSA key for encryption with JWK, PKCS7, PKCS11, and gpg.
How you encrypt the individual layer is irrelevant for this attack since the attack works by modifying one of the unencrypted layers.
How you encrypt the individual layer is irrelevant for this attack since the attack works by modifying one of the unencrypted layers.
How do you get to decrypt the encrypted layers if you don't have the RSA private key?
@lumjjb Do you refuse to decrypt if the manifest is unsigned?
That is the choice of the runtime policy, the encryption technology is agnostic to this, depending on the setup
That seems to me like the wrong choice to preserve the security of the system as a whole. If you enable an encrypted layer, you should prevent decryption whenever integrity of the whole cannot be verified.
The integrity of the whole can be verified with (previous) Notary or whatever may be in use these days -- we didn't want to re-invent this part. We do NOT guarantee that the unencrypted layers are not containing garbage but I would like to see how one can decrypt the RSA-encrypted layers.
That seems to me like the wrong choice to preserve the security of the system as a whole. If you enable an encrypted layer, you should prevent decryption whenever integrity of the whole cannot be verified.
My personal recommendation is to sign/verify images always. But I would defer this discussion to the orchestration components (i.e. kubernetes, cri-o, etc.), they are the decision makers in these matters.
How do you get to decrypt the encrypted layers if you don't have the RSA private key?
Under this attack, the attacker has access to:
- one of the unencrypted layers
- the unsigned manifest
The goal of the attacker is to extract the plaintext from the encrypted layer. However, the attacker does not have the key to perform decryption. Therefore, the attacker wants to trick the runtime that deploys the image into disclosing the plaintext on his behalf.
The attacker modifies the unencrypted layer to add a backdoor to an executable within the execution path (for example: bash). The attacker additionally modifies the manifest to contain the measurement of the backdoored layer.
When the container is deployed, the attacker does not have access to the deployment system. However, the deployment system validates the layer measurements (including the fraudulent measurement from the modified manifest) and assembles a container that contains both the backdoored binary (i.e. bash) and the plaintext from the encrypted layer. When the deployment system executes the container, the backdoored binary exfiltrates the plaintext.
Therefore, the confidentiality of the encrypted layer fails by modifying an unencrypted layer when there is no signature on the manifest. To defend against this attack, the deployment host must refuse to decrypt data whenever the manifest has no integrity protection (i.e. signature).
My personal recommendation is to sign/verify images always. But I would defer this discussion to the orchestration components (i.e. kubernetes, cri-o, etc.), they are the decision makers in these matters.
That requirement is probably onerous. Many people are happy to deploy images with no integrity/encryption. But, as I outlined above, you should refuse to decrypt an encrypted layer if there is no integrity protection on the manifest (or if there is some way to assemble a container from a layer with no measurement). This is because if you don't have integrity of all layers, you don't have confidentiality of any particular layer. Since the party that produce the container obviously wants confidentiality of a layer, a failure to sign the manifest in the presence of an encrypted layer should be treated as a misconfiguration that produces a security vulnerability.
An alternate implementation would include the measurements of each dependent layer inside the AEAD of the encrypted layer.
@npmccallum I don't disagree with your point that confidentiality != integrity. But I think that that is handled on a higher level, the proposal here is just talking about the mechanics of encryption, not the big picture. Kind of like DH is part of TLS or IPSec.
The audience for your problem is really towards the orchestration. So i'm going to tag a couple folks that will be able to provide more concrete actionable statements. I would recommend also bringing this to SIG-Node or a relevant kubernetes sig, and would be happy to be involved with the discussions there.
@mtrmac @mrunalp @mikebrow
An alternate implementation would include the measurements of each dependent layer inside the AEAD of the encrypted layer.
Would it have to be each dependent layer or if it makes it easy an 'accumulation' of the hashes of each of the previous layers? So like TPM PCR measurements: next_hash=sha256(cur_hash || layer_hash). That should work as well, or not?
@lumjjb My point is not that confidentiality != integrity. My point is that you are assembling an executable from parts, some of which are supposed to be secret. But if you allow an attacker to modify other parts of your executable and then combine the plaintext with it, you are unable to guarantee the privacy of the plaintext. This is not an orchestration problem. It is either a mechanics implementation problem or a protocol problem.