Sign local container images before they are pushed to registries

[yizha1]

Is your feature request related to a problem?

Background

Nowadays, in order to sign container images, users must first push the images to a registry. However, registries are not within the trust boundary of Notation, as outlined in the threat model. If a registry is compromised, attackers can tamper with container images during the short period between when they are pushed to the registry and when they are signed by users. From the perspective of image consumers, the image can be verified against the signature successfully, but it may still be malicious. This is a serious security issue that cannot be easily detected.

Notation supports an experimental feature that allows users to sign a local image in OCI layout format. However, this is not how most users build their container images. Most users build container images using Docker Engine, including Docker CLI and dockerd as the daemon. The container images built on a local machine are not in the format of OCI layout; instead, they are in Docker’s own format. Docker Engine also supports containerd as the daemon, and the image built on a local machine is in the format of OCI layout with minor differences. However, supporting containerd is not yet a stable feature.

Scenario

A software engineer wants to sign local container images built by docker build before pushing images to a registry. Verification of the image should be the same workflow for signing and verifying images stored in OCI complaint registry.

What solution do you propose?

docker save command can save a local image into a tar file, however, it is in the format of docker own, not OCI layout or other standard format. The tar file is not the one that docker push command pushes to the registry. So, if we sign local images created by docker save command, the digest is different from digest of manifest of the image that is pushed to the container registry. To sign tar file created by docker save, and verify it before docker load, I would suggest considering feature sign and verify arbitrary file.

My proposal is to create an official docker plugin to generate an OCI descriptor and sign the descriptor. Here is the experience.

1. Built a container image using command docker build/buildx docker build -t localhost:5001/net-monitor:v1 https://github.com/wabbit-networks/net-monitor.git#main
2. List the container images on local machine to confirm the image was built docker images
3. Run docker plugin named descriptor to generate a OCI descriptor for the container image. docker descriptor generate localhost:5001/net-monitor:v1 -o net-monitor.json
4. Sign the descriptor notation sign --descriptor net-monitor.json --output-signature digest.sig
5. Push the local container image to an OCI compliant registry docker push localhost:5001/net-monitor:v1
6. Push the signature notation push digest.sig localhost:5001/net-monitor:v1
7. List the signature to confirm it is existing and associated with the image notation list

What alternatives have you considered?

Use a tool to convert local docker images to an OCI layout, and then sign the OCI layout as the experimental feature.

Any additional context?

No response

[sajayantony]

why couldn’t we standardizing in OCI tar output from buildx?

[FeynmanZhou]

Per investigation in the Sign local images PRD and discussion in the community meeting on Oct 31, we chose to defer this feature to future releases since it has external dependencies. So I updated the milestone from v1.1.0 to future. We will need to prepare another proposal for integration with other image build tools (e.g. Docker buildx)

What we can improve in Notation v1.1.0 for the existing feature of signing local image is to update the user guide to simplify the prerequisite steps.

[tuminoid]

I see milestone is still set to "future", so this is not going to be addressed in v1.2.0 for example? Can you also elaborate what the "external dependencies" mentioned are, and if there is expected timeline for solving those?

[FeynmanZhou]

Hi @tuminoid ,

Thanks for your interest in this feature plan. The "external dependencies" actually means there will be some work that needs to be completed outside the Notary Project. For example, we want to integrate Notation into other image build tools (e.g. Docker buildx) to deliver the native signing experience in the image build process.

Are you looking for a local signing solution in Notation?

[tuminoid]

Hi @FeynmanZhou. We consider it both convenience feature (pushing, signing, pulling vs signing) and also as mentioned in the description, a security gap. There is workaround via --oci-layout but its experimental.