Anchore analysis fails on Redhat UBI Micro multi-stage build

[ssthom]

Is this a request for help?: Yes

---

Is this a BUG REPORT or a FEATURE REQUEST? (choose one): Bug Report

Version of Anchore Engine and Anchore CLI if applicable: $ anchore-cli --version anchore-cli, version 0.9.1

Engine DB Version: 0.0.14 Engine Code Version: 0.9.4

What happened:

Image analysis fails every time for this image

What did you expect to happen: Image should be scanned successfully scanned and vulnerability data able to be retreived

Any relevant log output from /var/log/anchore:

[service:worker] 2021-06-10 12:25:25+0000 [-] [Thread-8] [anchore_engine.services.analyzer.service/build_task()] [INFO] Starting image analysis thread
[service:worker] 2021-06-10 12:25:25+0000 [-] [Thread-258757] [anchore_engine.services.analyzer.analysis/process_analyzer_job()] [INFO] image dequeued for analysis: cicd : sha256:02043fb5ac93b643b691bebf2f2b66436ad55106c7937b931435f168b543c785
[service:worker] 2021-06-10 12:25:25+0000 [-] [Thread-258757] [anchore_engine.services.analyzer.analysis/perform_analyze()] [INFO] performing analysis on image: ['cicd', 'sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro@sha256:02043fb5ac93b643b691bebf2f2b66436ad55106c7937b931435f168b543c785', 'sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro:20210610-122219']
[service:worker] 2021-06-10 12:25:25+0000 [-] [Thread-258757] [anchore_engine.services.analyzer.analysis/perform_analyze()] [INFO] analyzing image: sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro@sha256:02043fb5ac93b643b691bebf2f2b66436ad55106c7937b931435f168b543c785
[service:worker] 2021-06-10 12:25:25+0000 [-] [Thread-258757] [anchore_engine.clients.localanchore_standalone/pull_image()] [INFO] Downloading image sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro@sha256:02043fb5ac93b643b691bebf2f2b66436ad55106c7937b931435f168b543c785 for analysis to /analysis_scratch/d230e190-9953-4f5c-884f-cc935f11ede8/raw
[service:worker] 2021-06-10 12:25:27+0000 [-] "172.30.166.108" - - [10/Jun/2021:12:25:26 +0000] "GET /metrics HTTP/1.1" 200 10022 "-" "Prometheus/2.22.1"
[service:worker] 2021-06-10 12:25:29+0000 [-] Traceback (most recent call last):
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 1092, in analyze_image
[service:worker] 2021-06-10 12:25:29+0000 [-]     imageSize = unpack(staging_dirs, layers)
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 876, in unpack
[service:worker] 2021-06-10 12:25:29+0000 [-]     squashtar, imageSize = squash(unpackdir, cachedir, layers)
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 330, in squash
[service:worker] 2021-06-10 12:25:29+0000 [-]     tarfiles[l] = tarfile.open(layertar, mode="r", format=tarfile.PAX_FORMAT)
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/lib64/python3.8/tarfile.py", line 1590, in open
[service:worker] 2021-06-10 12:25:29+0000 [-]     raise ValueError("nothing to open")
[service:worker] 2021-06-10 12:25:29+0000 [-] ValueError: nothing to open
[service:worker] 2021-06-10 12:25:29+0000 [-]
[service:worker] 2021-06-10 12:25:29+0000 [-] During handling of the above exception, another exception occurred:
[service:worker] 2021-06-10 12:25:29+0000 [-]
[service:worker] 2021-06-10 12:25:29+0000 [-] Traceback (most recent call last):
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/analyzer/analysis.py", line 333, in process_analyzer_job
[service:worker] 2021-06-10 12:25:29+0000 [-]     image_data = perform_analyze(
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/analyzer/analysis.py", line 193, in perform_analyze
[service:worker] 2021-06-10 12:25:29+0000 [-]     analyzed_image_report, manifest_raw = localanchore_standalone.analyze_image(
[service:worker] 2021-06-10 12:25:29+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 1125, in analyze_image
[service:worker] 2021-06-10 12:25:29+0000 [-]     raise AnalysisError(
[service:worker] 2021-06-10 12:25:29+0000 [-] anchore_engine.clients.localanchore_standalone.AnalysisError: failed to download, unpack, analyze, and generate image export (sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro@sha256:02043fb5ac93b643b691bebf2f2b66436ad55106c7937b931435f168b543c785) - exception: nothing to open
[service:worker] 2021-06-10 12:25:29+0000 [-] [Thread-258757] [anchore_engine.services.analyzer.analysis/process_analyzer_job()] [ERROR] problem analyzing image - exception: failed to download, unpack, analyze, and generate image export (sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro@sha256:02043fb5ac93b643b691bebf2f2b66436ad55106c7937b931435f168b543c785) - exception: nothing to open
[service:worker] 2021-06-10 12:25:30+0000 [-] [Thread-8] [anchore_engine.services.analyzer.service/handle_image_analyzer()] [INFO] worker thread completed

What docker images are you using:

Dockerfile

FROM registry.access.redhat.com/ubi8-minimal:latest as builder

RUN echo '[INFO] Upgrading & installing packages ...' \
    && microdnf update -y \
    && microdnf install -y shadow-utils \
    && echo '[INFO] Adding Sterling user ...' \
    && groupadd -g 1001 sterling \
    && useradd -u 1001 -r -g 1001 -m -s /bin/bash -c "Sterling User" sterling

FROM registry.access.redhat.com/ubi8-micro:latest

# Copy Linux user/group files
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group

# Create sterling directories
RUN mkdir -p /home/sterling \
  && chmod 755 /home/sterling && chown -R 1001:1001 /home/sterling \
  && mkdir -p /var/log/sterling \
  && chown -R 1001:1001 /var/log/sterling

# Default user/group
USER sterling:sterling

How to reproduce the issue:

1. Build the docker image above
2. Push image to a docker repository
3. Add the image to Anchore
4. Wait for image to fail analysis

Anything else we need to know:

It might be related to it being the micro image registry.access.redhat.com/ubi8-minimal:latest being very basic (No package manager, tar, curl, etc) but that image works if you scan it directly

[nurmi]

some additional context on this finding, as an attempt to reproduce by building/pushing an image with the provided Dockerfile was unsuccessful (in that the resulting image would analyze normally, without generating the error). Working with user, we did find that in the particular registry being used, there is a difference between the layers listed in the manifest versus the layers listed in the result of the skopeo download (oci directory), for just one of the layers. namely, the manifest has:

"layers" : [
      {
         "digest" : "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
         "mediaType" : "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size" : 1024
      },

where the oci manifest shows:

   "layers" : [
      {
         "digest" : "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3",
         "mediaType" : "application/vnd.oci.image.layer.v1.tar+gzip",
         "size" : 42
      },

for the first layer. this causes anchore to be unable to 'find' the first layer, as the code is using the layer digest from the manifest to look for the corresponding layer file post download. Details of the layer file itself show that the oci manifest entry lines up with the local layer file, but the digest from the manifest is against the uncompressed data, even though the mediaType is indicating that the layer is compressed:

% ls -l 4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3 
-rw-r--r--@ 1 nurmi  staff  42 Jun 15 12:37 4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3
% shasum -a 256 4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3 
4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3 
% cat 4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3| gunzip > uncompressed
% shasum -a 256 uncompressed 
5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef

Adding the above to aid with triage - we don't have an example of an image in a registry (that is public) that demonstrates this effect, but the above does explain the error in the logs.

Suggest considering adding a codepath for inspecting the oci manifest instead of (or in addition to, if there is a misalignment detected) the registry manifest, to handle this condition (one idea that could ensure that when this condition is present for a given registry/image, the analyzer would be able to find the layer on disk after download even if the manifest layer digest differs from the layer file that is ultimately present).

[nurmi]

More context - anchore uses skopeo to download image for analysis, with a command like:

skopeo   copy --remove-signatures --src-tls-verify=true --src-creds "${SKOPUSER}":"${SKOPPASS}"  docker://docker.io/some/repo@sha256:79fe86c297a8503c2504d9c3669d915d7de5c5b71901abcb07c438998d9c0bd9 oci:/tmp/foobar/raw:image

In the case discussed here, we observe that adding the skopeo copy option '--dest-oci-accept-uncompressed-layers' appears to result in the local layer file names matching the layer digests in the original manifest (adding this for consideration for a possible alternate solution).

Finally - the image in question is being pushed to a registry using skopeo, with signing

skopeo copy --debug --retry-times 3 --dest-creds ****:**** --sign-by **** docker-daemon:<local image> docker://<registry>/<repo>:<tag>

which may be a way to try and reproduce the effect (manifest layer digests not lining up with downloaded layer file names).

[zburstein]

This appears to be an identical issue to https://github.com/anchore/anchore-engine/issues/1080. @tsaarni did some investigation and provided a great writeup detailing the cause of the issue and a potential solution