ko SBOM can't be pushed to quay.io

SBOM can't be pushed to quay.io

Open imjasonh opened this issue 2 years ago • 12 comments

$ ko version
v0.9.4-0.20211208142815-ad0607f0a1eb

$ KO_DOCKER_REPO=quay.io/imjasonh ko build ./
2021/12/13 14:45:40 Using base golang:1.17 for github.com/google/ko
2021/12/13 14:45:41 Building github.com/google/ko for linux/amd64
2021/12/13 14:46:56 Publishing quay.io/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e:latest
2021/12/13 14:46:57 pushed blob: sha256:e4a33f5890d928895f7523a8d66d79aa990509a633e6b81d0e0006a55725f13f
2021/12/13 14:46:59 pushed blob: sha256:1ce30adc04063fdbfb60f34e83f3ec62e91c03e4b310586239902c782f4eac7c
Error: failed to publish images: error publishing ko://github.com/google/ko: writing sbom: PUT https://quay.io/v2/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e/manifests/sha256-7c4c064d8ca7880b4a98674201b2c5e7c6df7fb536b5f4a470acce1a9f75f14b.sbom: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]
2021/12/13 14:46:59 error during command execution:failed to publish images: error publishing ko://github.com/google/ko: writing sbom: PUT https://quay.io/v2/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e/manifests/sha256-7c4c064d8ca7880b4a98674201b2c5e7c6df7fb536b5f4a470acce1a9f75f14b.sbom: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]

If SBOM publishing was opt-in I'd say this is WAI and fine, but it's going to be a pretty bad experience for happy ko users who push to Quay (and other similar registries) when they upgrade to the next release.

Should we log-and-continue if SBOM publishing fails?

@mattmoor

Dec 13 '21 14:12 imjasonh

Yeah, this should be a release blocker. Perhaps we should use legacy media types until they get their act together?

cc @dlorenc is this the upstream guidance or what?

Dec 13 '21 16:12 mattmoor

cc @dlorenc is this the upstream guidance or what?

What do you mean by upstream? Sigstore?

I think I'd proceed with a warning here or make it opt-in. I don't think blocking a release is really doable given there's no ETA on quay working correctly.

Dec 13 '21 16:12 dlorenc

I just mean not failing outright. I think we either change the media type we use, or warn.

Dec 13 '21 16:12 mattmoor

+1 to log-and-continue if pushing the SBOM fails. This will also affect signatures, etc.

Dec 13 '21 16:12 imjasonh

I think you have to go all the way back to docker media types, including lying about the layer contents.

Dec 13 '21 17:12 dlorenc

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.

Mar 14 '22 01:03 github-actions[bot]

fyi: still getting that same error with latest ko on quay :

Error: error processing import paths in "config/452-gitlab.yaml": error resolving image references: writing sbom: PUT https://quay.io/v2/chmouel/pipelines-as-code/manifests/sha256-5eb0d8b1ff2da71cc9b07fe02b30e908b26859037bad9c4199ae95376e5509b3.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'text/spdx' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']

Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'enum': ['application/vnd.oci.image.layer.v1.tar',
              'application/vnd.oci.image.layer.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+zstd',
              'application/vnd.oci.image.layer.nondistributable.v1.tar',
              'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
              'application/vnd.dev.cosign.simplesigning.v1+json',
              'application/tar+gzip',
              'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+gzip'],
     'type': 'string'}

On instance['layers'][0]['mediaType']:
    'text/spdx']
2022/03/17 14:59:23 error during command execution:error processing import paths in "config/452-gitlab.yaml": error resolving image references: writing sbom: PUT https://quay.io/v2/chmouel/pipelines-as-code/manifests/sha256-5eb0d8b1ff2da71cc9b07fe02b30e908b26859037bad9c4199ae95376e5509b3.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'text/spdx' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']

Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'enum': ['application/vnd.oci.image.layer.v1.tar',
              'application/vnd.oci.image.layer.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+zstd',
              'application/vnd.oci.image.layer.nondistributable.v1.tar',
              'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
              'application/vnd.dev.cosign.simplesigning.v1+json',
              'application/tar+gzip',
              'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+gzip'],
     'type': 'string'}

On instance['layers'][0]['mediaType']:
    'text/spdx']

Mar 17 '22 14:03 chmouel

This is a known issue, add --sbom=none in the meantime until Quay.io is updated to accept these types by default. 🤷‍♂️

Mar 17 '22 18:03 imjasonh

Sounds Good, i guess we can keep this issue open until they do then!

Mar 21 '22 08:03 chmouel

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.

Jun 20 '22 01:06 github-actions[bot]

This might be available now? Does anybody feel like trying it out? 🙏

Jun 21 '22 13:06 imjasonh

still failing for me :

Jun 21 '22 18:06 chmouel

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.

Sep 20 '22 01:09 github-actions[bot]

Encountered this issue yesterday.

error: no objects passed to apply
Error: error processing import paths in "-": error resolving image references: writing sbom: PUT https://quay.io/v2/avinkuma/proxy-webhook-f8f95c9cea9508fe8915ae3d012d15fb/manifests/sha256-6e8f5b94738b4728ab479fad482ebbfaa2ac080b7feca17d5e28e4b2d15615b2.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/vnd.dsse.envelope.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'enum': ['application/vnd.oci.image.layer.v1.tar',
              'application/vnd.oci.image.layer.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+zstd',
              'application/vnd.oci.image.layer.nondistributable.v1.tar',
              'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
              'application/vnd.dev.cosign.simplesigning.v1+json',
              'application/vnd.dsse.envelope.v1+json',
              'application/tar+gzip',
              'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+gzip'],
     'type': 'string'}
On instance['layers'][0]['mediaType']:
    'spdx+json']
2022/09/21 12:56:31 error during command execution:error processing import paths in "-": error resolving image references: writing sbom: PUT https://quay.io/v2/avinkuma/proxy-webhook-f8f95c9cea9508fe8915ae3d012d15fb/manifests/sha256-6e8f5b94738b4728ab479fad482ebbfaa2ac080b7feca17d5e28e4b2d15615b2.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/vnd.dsse.envelope.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'enum': ['application/vnd.oci.image.layer.v1.tar',
              'application/vnd.oci.image.layer.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+zstd',
              'application/vnd.oci.image.layer.nondistributable.v1.tar',
              'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
              'application/vnd.dev.cosign.simplesigning.v1+json',
              'application/vnd.dsse.envelope.v1+json',
              'application/tar+gzip',
              'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+gzip'],
     'type': 'string'}
On instance['layers'][0]['mediaType']:
    'spdx+json']
make: *** [Makefile:111: apply] Error 1

Adding --sbom=none works.

Sep 22 '22 06:09 avinal

I'm going to close this issue, since I'm not sure there's a lot we can do until quay.io supports these types.

Sep 22 '22 15:09 imjasonh

@imjasonh Quay.io product manager here. We can certainly look into adding support for the missing types. What specifically does ko expect here?

Oct 11 '22 09:10 dmesser

Hi @dmesser, as mentioned here https://github.com/ko-build/ko/blob/e1b4eade08febafdb9c35f11b54b699e5c04c5e5/docs/features/sboms.md?plain=1#L6 it seems quay yet not have support for SBOM for the pushed image. So pushing sbom in any format (spdx or cyclonedx) fails.

Oct 11 '22 14:10 avinal

@avinal SPDX and CycloneDX themselves are known formats. The OCI type which is used to ship those SBOMs as an OCI artifact however is not standardized, it depends on the client that is used. SigStore seems to use these here: https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md#mediatypes

We are looking to enabling these shortly. Hence the question if ko aligns with that or has separate type expectations. OCI types are basically mediaType properties that need to conform to RFC 6838. There is no finite and standard list of distinct values though.

Oct 11 '22 14:10 dmesser

Referring to the implementation here: https://github.com/ko-build/ko/tree/main/internal/sbom only these two types are expected by ko. I would let @imjasonh explain this more precisely.

Oct 11 '22 14:10 avinal

Hey @dmesser 👋

ko uses the sigstore-specified media types when pushing SBOMs. Previous versions of ko pushed text/spdx, but now push text/spdx+json, and IIRC push application/vnd.cyclonedx+json as well.

If the whole list of media types in sigstore's SBOM_SPEC.md were supported by Quay that would be great, and we could be free to choose which types+variants we support over time (e.g., maybe eventually add syft SBOMs).

Let me know if there's anything else I can help with!

Oct 11 '22 15:10 imjasonh

@imjasonh sounds good. Yes, we are looking to enable the full list from SigStore.

Oct 11 '22 15:10 dmesser

FYI https://twitter.com/quayio/status/1580121221080891392

Oct 13 '22 17:10 dmesser

But still is doesn´t work (yet):

2023/03/05 21:05:03 Publishing quay.io/pamvdam/containers-57bcbf1d2df6010d41816c012905835d:latest
2023/03/05 21:05:04 existing blob: sha256:21219c3231f5a1cca4febbaf40f698b78e04a2d2860b65346ccd1c7db0ef2e92
2023/03/05 21:05:05 pushed blob: sha256:b339feae942b1dcc6edb5974d16f82c7c719464e6b41bf2561c1800c57291a06
Error: failed to publish images: error publishing ko://containers: writing sbom: PUT https://quay.io/v2/pamvdam/containers-57bcbf1d2df6010d41816c012905835d/manifests/sha256-368d87ad02e96d670654eddf4053c5b2139f3492776ea8f2ae5fdd06c281ada7.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' does not match '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?'

Failed validating 'pattern' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'pattern': '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?',
     'type': 'string'}

On instance['layers'][0]['mediaType']:
    'spdx+json']
2023/03/05 21:05:06 error during command execution:failed to publish images: error publishing ko://containers: writing sbom: PUT https://quay.io/v2/pamvdam/containers-57bcbf1d2df6010d41816c012905835d/manifests/sha256-368d87ad02e96d670654eddf4053c5b2139f3492776ea8f2ae5fdd06c281ada7.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' does not match '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?'

Failed validating 'pattern' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'pattern': '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?',
     'type': 'string'}

On instance['layers'][0]['mediaType']:
    'spdx+json']

Mar 05 '23 20:03 pascal71

This might be fixed in ko at head: https://github.com/ko-build/ko/issues/970#issuecomment-1456951250

Mar 07 '23 17:03 imjasonh

ko ko copied to clipboard

SBOM can't be pushed to quay.io

ko
ko copied to clipboard