dockle stopped working when updating docker to version 25

[tgquan67]

Description

Since I updated docker engine to version 25, I cannot scan images built locally anymore, with or without buildkit. The image is built normally with a dockerfile

docker build . -t a:b --load

And scan with

dockle a:b

This happens on both my local machine with Ubuntu 20.04 and dockle 0.4.11 and my Jenkins system with Ubuntu 20.04 and dockle 0.4.14. I already tried setting DOCKLE_HOST but that did not solve the problem.

What did you expect to happen? The image is scanned.

What happened instead?

2024-03-05T15:55:55.182+0200    FATAL   unable to initialize a image struct: failed to initialize source: reading manifest b in docker.io/library/a: requested access to the resource is denied

Output of run with -debug:

2024-03-05T15:56:01.945+0200    DEBUG   There is no .dockleignore file
2024-03-05T15:56:01.945+0200    DEBUG   Skipped update confirmation
2024-03-05T15:56:01.945+0200    DEBUG   Start assessments...
2024-03-05T15:56:03.669+0200    FATAL   unable to initialize a image struct:
    github.com/goodwithtech/deckoder/extractor/docker.newDockerExtractor
        /home/runner/go/pkg/mod/github.com/goodwithtech/[email protected]/extractor/docker/docker.go:73
  - failed to initialize source:
    github.com/goodwithtech/deckoder/extractor/image.NewImage
        /home/runner/go/pkg/mod/github.com/goodwithtech/[email protected]/extractor/image/image.go:86
  - reading manifest b in docker.io/library/a: requested access to the resource is denied

Output of dockle -v:

dockle version 0.4.11

Additional details (base image name, container registry info...): Sample dockerfile used

FROM python:3.8
COPY Pipfile Pipfile.lock .
RUN pip install pipenv && \
    pipenv sync
COPY code.txt .

[lamroger-nava]

I'm having the same issue - did you figure out the problem @tgquan67 ?

edit: I figured it out - the runner / docker was out of space. I run this before checkout

- name: Delete huge unnecessary tools folder
        run: rm -rf /opt/hostedtoolcache

[tgquan67]

As far as I can see, the problem seems to have been solved with docker 26, not sure what docker did though.

[BertelBB]

Also experiencing this.

Docker version 26.1.3, build b72abbb

Seems that Dockle is no longer able to scan local images. Always tries to pull from DockerHub, even if no user is set in the image tag. E.g. app-name:latest. I think historically, Docker would assume images without a registry or docker-hub-user/ in the image name were local images. Now it always tries to pull...

[phyzical]

also just started running into this due to i think the docker version being updated on github runnners base image? i dont have this issue on custom runners atm

edit: yeah looks like the ubuntu-latest bumped the docker server version from 24 to 26 last night https://github.com/actions/runner-images/blame/main/images/ubuntu/Ubuntu2204-Readme.md#L82

Why its behaving this why idk... maybe something buildkit related now that 26 tries to push it harder?

[BertelBB]

I managed to solve this by using the --load flag when I build the container image. I could not get it to work when using goodwithtech/[email protected]. Not sure if that is because it uses [email protected] or something else. Instead of using the action, I simply run Dockle as a container.

This is what I am doing in a nutshell:

steps:
- name: Checkout
  uses: actions/checkout@v4

- name: Setup Docker buildx
  uses: docker/setup-buildx-action@v3

- name: Docker build
  run: |
    docker buildx build -t app:latest --load ./src

- name: Lint container using Dockle
  uses: docker://goodwithtech/dockle:v0.4.14
  with:
    args: app:latest

[tgquan67]

In @BertelBB's case, it's not related to dockle, it's just how docker buildkit works (which was enabled by default since docker 23). Basically when you build with buildkit, the result image will remain in cache and will not be exported to docker daemon (where dockle tries to find the image) unless you explicitly tell it so with --load. It's mostly related to the builder you use, but I have seen other people who use almost the same builder as mine but doesn't need --load, so the behavior is a bit inconsistent, and I haven't been able to track down the exact difference.

[BertelBB]

In @BertelBB's case, it's not related to dockle, it's just how docker buildkit works (which was enabled by default since docker 23). Basically when you build with buildkit, the result image will remain in cache and will not be exported to docker daemon (where dockle tries to find the image) unless you explicitly tell it so with --load. It's mostly related to the builder you use, but I have seen other people who use almost the same builder as mine but doesn't need --load, so the behavior is a bit inconsistent, and I haven't been able to track down the exact difference.

EDIT: I spoke too soon in my previous version of this comment. I am able to replicate this locally when using [email protected], even if I build using the --load flag.

In GH Actions, I had to use the --load flag and [email protected]. I tried both docker://goodwithtech/dockle:v0.4.13 and goodwithtech/[email protected] (latest action version, targets [email protected]) and it failed in both cases.

[tgquan67]

@BertelBB can you confirm that after you build, you can find your new image in the output of docker images?

[tgquan67]

Update: I can confirm that docker 26.1 is not working with dockle again:

$ docker images
REPOSITORY      TAG               IMAGE ID       CREATED              SIZE
a               b                 a49f44b85bbb   About a minute ago   77.9MB
moby/buildkit   buildx-stable-1   480495983c47   4 months ago         172MB

$ dockle a:b
2024-06-14T14:15:53.212+0300    FATAL   unable to initialize a image struct: failed to initialize source: reading manifest b in docker.io/library/a: requested access to the resource is denied

$ dockle -v
dockle version 0.4.11

$ docker --version
Docker version 26.1.3, build b72abbb

It worked on docker 24.0.8 but not 25, and iirc it also worked on docker 26.0. Try to downgrade docker to version 26.0.2 and test again.

[BertelBB]

I see the image when I do docker images, regardless of if I use --load flag or not.

$ docker --version
Docker version 26.1.4, build 5650f9b

$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

# Build without --load flag
$ docker buildx build --tag sample-service:latest ./src

$ docker images
REPOSITORY       TAG       IMAGE ID       CREATED         SIZE
sample-service   latest    3bb79129cf96   7 seconds ago   204MB

$ dockle --version
dockle version 0.4.14

# Scan with [email protected] on image built without --load flag
$ dockle sample-service:latest
WARN    - DKL-DI-0006: Avoid latest tag
        * Avoid 'latest' tag
INFO    - CIS-DI-0005: Enable Content trust for Docker
        * export DOCKER_CONTENT_TRUST=1 before docker pull/build
INFO    - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
        * not found HEALTHCHECK statement

# Downgrade Dockle to 0.4.13
$ curl -L -o dockle.deb https://github.com/goodwithtech/dockle/releases/download/v0.4.13/dockle_0.4.13_Linux-64bit.deb
$ sudo dpkg -i dockle.deb

$ dockle --version
dockle version 0.4.13

# Scan with [email protected] on image build without --load flag
$ dockle sample-service:latest
2024-06-14T11:49:31.163Z        FATAL   unable to initialize a image struct: failed to initialize source: reading manifest latest in docker.io/library/sample-service: requested access to the resource is denied

# Cleanup
$ docker system prune -fa

$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

# Build with --load flag
$ docker buildx build --load --tag sample-service:latest ./src

$ docker images
REPOSITORY       TAG       IMAGE ID       CREATED         SIZE
sample-service   latest    15a358e41a3f   2 seconds ago   204MB

# Scan with [email protected] on image built with --load flag
$ dockle sample-service:latest
2024-06-14T11:52:14.955Z        FATAL   unable to initialize a image struct: failed to initialize source: reading manifest latest in docker.io/library/sample-service: requested access to the resource is denied

# Upgrade Dockle to v0.4.14
$ curl -L -o dockle.deb https://github.com/goodwithtech/dockle/releases/download/v0.4.14/dockle_0.4.14_Linux-64bit.deb
$ sudo dpkg -i dockle.deb

# dockle --version
$ dockle version 0.4.14

# Scan with [email protected] on image built with --load flag
$ dockle sample-service:latest
WARN    - DKL-DI-0006: Avoid latest tag
        * Avoid 'latest' tag
INFO    - CIS-DI-0005: Enable Content trust for Docker
        * export DOCKER_CONTENT_TRUST=1 before docker pull/build
INFO    - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
        * not found HEALTHCHECK statement

As you can see from the above, using the --load flag doesn't seem to matter.

• [email protected] on image built without --load flag: ❌ Fails
• [email protected] on image built without --load flag: ✅ Succeeds
• [email protected] on image built with --load flag: ❌ Fails
• [email protected] on image built with --load flag: ✅ Succeeds

In GH Actions however, I had to use --load flag to get it to work...

[tgquan67]

@BertelBB then I guess --load doesn't matter to this particular issue, just use it when you can't get your image to docker images. It seems like dockle version 0.4.14 is working for you, however on Ubuntu 20.04 the maximum version I can get from apt repo is 0.4.11, so I will try to install a newer version from deb package later

[cpannwitz]

Are there any updates, or working workarounds, to this situation?

Our Github Action looks a bit like this, and fails with the before mentioned error:

      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: build local container
        uses: docker/build-push-action@v5
        with:
          tags: IMAGENAME:latest
          push: false
          load: true

      - name: Run dockle
        uses: goodwithtech/dockle-action@main
        env:
          DOCKLE_HOST: 'unix:///var/run/docker.sock'

        with:
          image: 'IMAGENAME:latest'
          format: 'list'
          exit-code: '1'
          exit-level: 'warn'

[tgquan67]

@cpannwitz did you try @BertelBB's solution above? https://github.com/goodwithtech/dockle/issues/256#issuecomment-2166137411 and https://github.com/goodwithtech/dockle/issues/256#issuecomment-2167592655

[cpannwitz]

Yeah, tried it. Got it to work, essentially by not using the Github Action, but the Docker Container instead.

- name: Lint container using Dockle
        uses: docker://goodwithtech/dockle:v0.4.14
        with:
          args: '--exit-code 1 --exit-level warn --format list IMAGENAME:latest'

[faramat]

any ideas?

[phyzical]

using load resolved it for me too

But i also moved our pipeline over to using the build-push action + ghcr for intermediate storage instead of cli buildx + local refs.

so its possible im getting around the issue as its just pulling from the ghcr instead?