Fix DCT Security Theater (TOFU)

[maltfield]

Description

In my setup, Docker Content Trust is 100% security theater. It provides zero security because of TOFU.

Steps to reproduce the issue:

1. Launch a new (ephermal) build server
2. Download image from docker hub ...
3. Build Finishes, ephemeral build server is destoryed
4. GOTO 1

Describe the results you received:

Every time I execute my CI build pipeline (which necessarily runs in a new instance on every run), docker downloads the image without telling me it was also downloading the key at the same time, which means it's actually not able to cryptographically verify the authenticity and integrity of the image.

root@disp9131:~# export DOCKER_CONTENT_TRUST=1
root@disp9131:~#

root@disp9131:~# ls $HOME/.docker/trust/tuf/docker.io/library
debian
root@disp9131:~# 

root@disp9131:~# docker pull ubuntu:latest
Pull (1 of 1): ubuntu:latest@sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537
sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537: Pulling from library/ubuntu
d72e567cc804: Pull complete 
0f3630e5ff08: Pull complete 
b6a83d81d1f4: Pull complete 
Digest: sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537
Status: Downloaded newer image for ubuntu@sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537
Tagging ubuntu@sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537 as ubuntu:latest
root@disp9131:~# 

root@disp9131:~# ls $HOME/.docker/trust/tuf/docker.io/library
debian  ubuntu
root@disp9131:~# 

Describe the results you expected:

By default, if running docker pull and the root key is not already downloaded, then the command should:

1. fail in non-interactive mode or

2. inform the user that the key isn't present and that it will have to download & TOFU the root key in order to proceed. Print a URL to a doc that provides more information on the risks in this, then prompt the user if they want to abort or continue with the tofu

It should not proceed as normal if the key is not present, misleading the user into thinking that their image download was cryptographically verified in a secure manner (when it wasn't).

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

root@disp9131:~# docker --version
Docker version 18.09.1, build 4c52b90
root@disp9131:~# 

Output of docker info:

root@disp9131:~# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.09.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 1.0.0~rc6+dfsg1-3
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.19.132-1.pvops.qubes.x86_64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.374GiB
Name: disp9131
ID: RTJT:KHJH:AW2N:AHCV:L4SD:3FYG:LIUV:FAYO:CMQM:LZX3:N5US:KULB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

root@disp9131:~# 

Additional environment details (AWS, VirtualBox, physical, etc.):

QubesOS -> Debian 10

See Also

1. https://security.stackexchange.com/questions/238529/how-to-list-all-of-the-known-root-keys-in-docker-docker-content-trust
2. https://github.com/BusKill/buskill-app/issues/6#issuecomment-700050760

[justincormack]

Yes, this is one of the design goals for Notary v2, to resolve this use case.

You can distribute the ~/.docker/trust directory to your build servers to provider initial keys.

[maltfield]

@justincormack Thank you. Can you please inform what is the minimum required files to pin the root public key used to sign a given set of images in the docker.io trust library? Just the root.json file or others as well?

Is this documented anywhere?

[ghost]

Hi @maltfield I have the same question too. So if I understand correctly, for now, we will have to manually create the whole folder structure (~/.docker/trust/tuf/XXXXX) and import the root.json files before pulling the image in order for DCT to use it?

[maltfield]

@zyitingftnt afaik, docker has not documented this anywhere.

I wrote-up the commands (with a verification) for how seed/pin a given docker publisher's public keys (trust initialization) here:

• https://devops.stackexchange.com/questions/13987/how-to-pin-public-root-key-when-downloading-an-image-with-docker-pull-docker-co/13988

[ghost]

@maltfield Thanks. I tried out your solution. So it's is not very convenient to manually create the directories and place the root.json, I guess this is the reason that people is working these 2 doc: https://docs.google.com/document/d/1JOBAlCDuf5JnpVLW54voGuAdsnMmSR2LIbs8fjBDSaM/edit#heading=h.lrzeqtpbqb2s https://docs.google.com/document/d/1Skt3d0twJjtNioyCaRN8ZF-vZZDT50Eg-F4QKdRKFcM/edit#

[maltfield]

@zyitingftnt those links are great! How did you find those docs??

[ghost]

@maltfield after 1 million hours in research LOL

[ghost]

@maltfield I think the solution is here if you use a notary client: https://github.com/notaryproject/notary/blob/master/docs/reference/client-config.md

[hilbix]

Trick to disable TOFU for unprivileged users (on Linux)

It seems to disables TOFU on nonprivileged users.

Sorry if this is already known, but I discovered it a few seconds ago on my own and did not find it here yet.

I am very new to Docker because I did not understand how to use it in a secure fashion. Today I had to learn about this security theater, because I needed to create my first VM which includes Docker (this not yet includes the fixes here). So I looked for a way to globally disable TOFU (and globally enforce DOCKER_CONTENT_TRUST, for me it looks like this can be done by wrapping the docker command with divert).

The trick seems to be to prevent the client from creating new root.json files within metadata/, as it seems to "properly fail" then.

Here is a script which does this:

mkdir -p ~/.docker/trust || :;

find ~/.docker/trust/. -type f -name root.json -print0 | xargs --null chmod 444 --;

find ~/.docker/trust/. -type d -print0 |
while read -rd '' dir;
do if [ -d "$dir/metadata" ]; then chmod 755 "$dir"; else chmod 555 "$dir"; fi; done;

I am not sure this works if docker is run from root, though, as I haven't tested it.

For the examples below I already had ~/.docker/trust primed with the library/debian key.

Now all we need to implement is a way to download the root.json files and verify them somehow. I, for my part, probably will just prime them using Ansible.

Example:

docker@docker:~$ echo $DOCKER_CONTENT_TRUST
1

docker@docker:~$ docker run hello-world
docker: error establishing connection to trust repository: mkdir /home/docker/.docker/trust/tuf/docker.io/library/hello-world: permission denied.
See 'docker run --help'.

But

docker@docker:~$ docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

docker@docker:~$ find ~/.docker/trust -ls
   158657      4 dr-xr-xr-x   4 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust
   158658      4 dr-xr-xr-x   3 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust/tuf
   158659      4 dr-xr-xr-x   3 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust/tuf/docker.io
   158660      4 dr-xr-xr-x   3 docker   docker       4096 Apr  7 16:49 /home/docker/.docker/trust/tuf/docker.io/library
   158655      4 drwxr-xr-x   3 docker   docker       4096 Apr  7 17:32 /home/docker/.docker/trust/tuf/docker.io/library/debian
   158663      4 dr-xr-xr-x   2 docker   docker       4096 Apr  7 16:53 /home/docker/.docker/trust/tuf/docker.io/library/debian/metadata
   158668      4 -r--r--r--   1 docker   docker       2393 Apr  7 16:53 /home/docker/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
   158664      4 dr-xr-xr-x   2 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust/private

docker@docker:~$ docker pull debian
Using default tag: latest
Pull (1 of 1): debian:latest@sha256:e97ee92bf1e11a2de654e9f3da827d8dce32b54e0490ac83bfc65c8706568116
docker.io/library/debian@sha256:e97ee92bf1e11a2de654e9f3da827d8dce32b54e0490ac83bfc65c8706568116: Pulling from library/debian
71215d55680c: Pull complete
Digest: sha256:e97ee92bf1e11a2de654e9f3da827d8dce32b54e0490ac83bfc65c8706568116
Status: Downloaded newer image for debian@sha256:e97ee92bf1e11a2de654e9f3da827d8dce32b54e0490ac83bfc65c8706568116
Tagging debian@sha256:e97ee92bf1e11a2de654e9f3da827d8dce32b54e0490ac83bfc65c8706568116 as debian:latest
docker.io/library/debian:latest

docker@docker:~$ find ~/.docker/trust -ls
   158657      4 dr-xr-xr-x   4 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust
   158658      4 dr-xr-xr-x   3 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust/tuf
   158659      4 dr-xr-xr-x   3 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust/tuf/docker.io
   158660      4 dr-xr-xr-x   3 docker   docker       4096 Apr  7 16:49 /home/docker/.docker/trust/tuf/docker.io/library
   158655      4 drwxr-xr-x   4 docker   docker       4096 Apr  7 17:32 /home/docker/.docker/trust/tuf/docker.io/library/debian
   158663      4 dr-xr-xr-x   2 docker   docker       4096 Apr  7 16:53 /home/docker/.docker/trust/tuf/docker.io/library/debian/metadata
   158668      4 -r--r--r--   1 docker   docker       2393 Apr  7 16:53 /home/docker/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
   158666      4 drwx------   2 docker   docker       4096 Apr  7 17:32 /home/docker/.docker/trust/tuf/docker.io/library/debian/changelist
   158664      4 dr-xr-xr-x   2 docker   docker       4096 Apr  7 16:14 /home/docker/.docker/trust/private

Note that changelist was added.

Looks like some crude, uncomfortable and unfriendly workaround for me, however for me it is better than nothing.

It does not prevent from using SHAs. AFAICS this is the right thing to do.

Compare:

docker@docker:~$ docker pull debian:buster
Pull (1 of 1): debian:buster@sha256:f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4
docker.io/library/debian@sha256:f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4: Pulling from library/debian
a9a7bf5145e4: Pull complete
Digest: sha256:f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4
Status: Downloaded newer image for debian@sha256:f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4
Tagging debian@sha256:f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4 as debian:buster
docker.io/library/debian:buster

docker@docker:~$ docker trust inspect debian:buster
[
    {
        "Name": "debian:buster",
        "SignedTags": [
            {
                "SignedTag": "buster",
                "Digest": "f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4",
                "Signers": [
                    "Repo Admin"
                ]
            }
        ],
        "Signers": [],
        "AdministrativeKeys": [
            {
                "Name": "Root",
                "Keys": [
                    {
                        "ID": "575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"
                    }
                ]
            },
            {
                "Name": "Repository",
                "Keys": [
                    {
                        "ID": "5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"
                    }
                ]
            }
        ]
    }
]

with:

docker@docker:~$ docker pull hello-world@sha256:e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57
docker.io/library/hello-world@sha256:e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57
Status: Downloaded newer image for hello-world@sha256:e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57
docker.io/library/hello-world@sha256:e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57

docker@docker:~$ docker trust inspect hello-world
[]
mkdir /home/docker/.docker/trust/tuf/docker.io/library/hello-world: permission denied