Some error messages from loading `provider-metadata.json` are dropped

[mgoetzegb]

If it is attempted to get the provider-metadata.json from https://csaf.data.security.<domain>" because the previous attempts from other locations failed, all error messages from the previous attempts are dropped.

The issue is in this line where the previous pmdl.messages are discarded because they are not added to the returned LoadedProviderMetadata object. See proposed fix in PR 531.

Steps to reproduce:

Create Dockerfile in repo root with content:

FROM golang:1.22.2-alpine  AS builder

ENV CGO_ENABLED=0

WORKDIR /src/

COPY go.mod .
COPY go.sum .
RUN go mod download

COPY cmd/csaf_downloader cmd/csaf_downloader
COPY csaf csaf
COPY internal internal
COPY util util


RUN go build -o ./bin/csaf_downloader ./cmd/csaf_downloader

FROM busybox
# image has no ca certs, all outgoing https connections will fail
COPY --from=builder /src/bin/csaf_downloader /bin/

ENTRYPOINT ["./bin/csaf_downloader"]

Build image and run download:

docker build -t csaf-dl-no-fix  .
docker run csaf-dl-no-fix --log_file "" --log_level debug sick.com

This will fail due to missing ca certificates in the image. The output is

2024/04/26 17:13:46 using STDERR for logging
{"time":"2024-04-26T15:13:46Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://sick.com/.well-known/csaf/provider-metadata.json"}
{"time":"2024-04-26T15:13:46Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://sick.com/.well-known/security.txt"}
{"time":"2024-04-26T15:13:46Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://sick.com/security.txt"}
{"time":"2024-04-26T15:13:47Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://csaf.data.security.sick.com"}
{"time":"2024-04-26T15:18:17Z","level":"DEBUG","msg":"Loading provider-metadata.json","domain":"sick.com","message":"fetching \"https://csaf.data.security.sick.com\" failed: Get \"https://csaf.data.security.sick.com\": dial tcp 203.0.113.1:443: connect: connection timed out"}
{"time":"2024-04-26T15:18:17Z","level":"INFO","msg":"Download statistics","succeeded":0,"total_failed":0,"filename_failed":0,"download_failed":0,"schema_failed":0,"remote_failed":0,"sha256_failed":0,"sha512_failed":0,"signature_failed":0}
{"time":"2024-04-26T15:18:17Z","level":"INFO","msg":"error: no valid provider-metadata.json found for 'sick.com'"}

The error message simply shows the error message from the last attempt to retrieve the provider-metadata.json via https://csaf.data.security.sick.com:

{"time":"2024-04-26T15:18:17Z","level":"DEBUG","msg":"Loading provider-metadata.json","domain":"sick.com","message":"fetching \"https://csaf.data.security.sick.com\" failed: Get \"https://csaf.data.security.sick.com\": dial tcp 203.0.113.1:443: connect: connection timed out"}

The actual error (missing ca certificates) is hidden, as all errors related to the previous attempts to access https://sick.com/.well-known/csaf/provider-metadata.json , https://sick.com/.well-known/security.txt and https://csaf.data.security.sick.com are dropped.

Same procedure with the fixed code gives the output with error messages from all attempts:

2024/04/26 17:26:29 using STDERR for logging
{"time":"2024-04-26T15:26:29Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://sick.com/.well-known/csaf/provider-metadata.json"}
{"time":"2024-04-26T15:26:30Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://sick.com/.well-known/security.txt"}
{"time":"2024-04-26T15:26:30Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://sick.com/security.txt"}
{"time":"2024-04-26T15:26:30Z","level":"DEBUG","msg":"http","who":"downloader","method":"GET","url":"https://csaf.data.security.sick.com"}
{"time":"2024-04-26T15:31:00Z","level":"DEBUG","msg":"Loading provider-metadata.json","domain":"sick.com","message":"fetching \"https://sick.com/.well-known/csaf/provider-metadata.json\" failed: Get \"https://sick.com/.well-known/csaf/provider-metadata.json\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"time":"2024-04-26T15:31:00Z","level":"DEBUG","msg":"Loading provider-metadata.json","domain":"sick.com","message":"Fetching \"https://sick.com/.well-known/security.txt\" failed: Get \"https://sick.com/.well-known/security.txt\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"time":"2024-04-26T15:31:00Z","level":"DEBUG","msg":"Loading provider-metadata.json","domain":"sick.com","message":"Fetching \"https://sick.com/security.txt\" failed: Get \"https://sick.com/security.txt\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"time":"2024-04-26T15:31:00Z","level":"DEBUG","msg":"Loading provider-metadata.json","domain":"sick.com","message":"fetching \"https://csaf.data.security.sick.com\" failed: Get \"https://csaf.data.security.sick.com\": dial tcp 203.0.113.1:443: connect: connection timed out"}
{"time":"2024-04-26T15:31:00Z","level":"INFO","msg":"Download statistics","succeeded":0,"total_failed":0,"filename_failed":0,"download_failed":0,"schema_failed":0,"remote_failed":0,"sha256_failed":0,"sha512_failed":0,"signature_failed":0}
{"time":"2024-04-26T15:31:00Z","level":"INFO","msg":"error: no valid provider-metadata.json found for 'sick.com'"}

[bernhardreiter]

@mgoetzegb thanks for the additional explanations, we will take a look. (But it may take a few days.)

[koplas]

Closed with #531