ko
ko copied to clipboard
SBOM can't be pushed to quay.io
$ ko version
v0.9.4-0.20211208142815-ad0607f0a1eb
$ KO_DOCKER_REPO=quay.io/imjasonh ko build ./
2021/12/13 14:45:40 Using base golang:1.17 for github.com/google/ko
2021/12/13 14:45:41 Building github.com/google/ko for linux/amd64
2021/12/13 14:46:56 Publishing quay.io/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e:latest
2021/12/13 14:46:57 pushed blob: sha256:e4a33f5890d928895f7523a8d66d79aa990509a633e6b81d0e0006a55725f13f
2021/12/13 14:46:59 pushed blob: sha256:1ce30adc04063fdbfb60f34e83f3ec62e91c03e4b310586239902c782f4eac7c
Error: failed to publish images: error publishing ko://github.com/google/ko: writing sbom: PUT https://quay.io/v2/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e/manifests/sha256-7c4c064d8ca7880b4a98674201b2c5e7c6df7fb536b5f4a470acce1a9f75f14b.sbom: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]
2021/12/13 14:46:59 error during command execution:failed to publish images: error publishing ko://github.com/google/ko: writing sbom: PUT https://quay.io/v2/imjasonh/ko-98b8c7facdad74510a7cae0cd368eb4e/manifests/sha256-7c4c064d8ca7880b4a98674201b2c5e7c6df7fb536b5f4a470acce1a9f75f14b.sbom: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]
If SBOM publishing was opt-in I'd say this is WAI and fine, but it's going to be a pretty bad experience for happy ko
users who push to Quay (and other similar registries) when they upgrade to the next release.
Should we log-and-continue if SBOM publishing fails?
@mattmoor
Yeah, this should be a release blocker. Perhaps we should use legacy media types until they get their act together?
cc @dlorenc is this the upstream guidance or what?
cc @dlorenc is this the upstream guidance or what?
What do you mean by upstream? Sigstore?
I think I'd proceed with a warning here or make it opt-in. I don't think blocking a release is really doable given there's no ETA on quay working correctly.
I just mean not failing outright. I think we either change the media type we use, or warn.
+1 to log-and-continue if pushing the SBOM fails. This will also affect signatures, etc.
I think you have to go all the way back to docker media types, including lying about the layer contents.
This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.
fyi: still getting that same error with latest ko on quay :
Error: error processing import paths in "config/452-gitlab.yaml": error resolving image references: writing sbom: PUT https://quay.io/v2/chmouel/pipelines-as-code/manifests/sha256-5eb0d8b1ff2da71cc9b07fe02b30e908b26859037bad9c4199ae95376e5509b3.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'text/spdx' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
{'description': 'The MIME type of the referenced manifest',
'enum': ['application/vnd.oci.image.layer.v1.tar',
'application/vnd.oci.image.layer.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+zstd',
'application/vnd.oci.image.layer.nondistributable.v1.tar',
'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
'application/vnd.dev.cosign.simplesigning.v1+json',
'application/tar+gzip',
'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+gzip'],
'type': 'string'}
On instance['layers'][0]['mediaType']:
'text/spdx']
2022/03/17 14:59:23 error during command execution:error processing import paths in "config/452-gitlab.yaml": error resolving image references: writing sbom: PUT https://quay.io/v2/chmouel/pipelines-as-code/manifests/sha256-5eb0d8b1ff2da71cc9b07fe02b30e908b26859037bad9c4199ae95376e5509b3.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'text/spdx' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
{'description': 'The MIME type of the referenced manifest',
'enum': ['application/vnd.oci.image.layer.v1.tar',
'application/vnd.oci.image.layer.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+zstd',
'application/vnd.oci.image.layer.nondistributable.v1.tar',
'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
'application/vnd.dev.cosign.simplesigning.v1+json',
'application/tar+gzip',
'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+gzip'],
'type': 'string'}
On instance['layers'][0]['mediaType']:
'text/spdx']
This is a known issue, add --sbom=none
in the meantime until Quay.io is updated to accept these types by default. 🤷♂️
Sounds Good, i guess we can keep this issue open until they do then!
This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.
This might be available now? Does anybody feel like trying it out? 🙏
still failing for me :
![image](https://user-images.githubusercontent.com/98980/174868054-e9891783-4bdb-4ff9-a9c4-5e94afc095d3.png)
This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.
Encountered this issue yesterday.
error: no objects passed to apply
Error: error processing import paths in "-": error resolving image references: writing sbom: PUT https://quay.io/v2/avinkuma/proxy-webhook-f8f95c9cea9508fe8915ae3d012d15fb/manifests/sha256-6e8f5b94738b4728ab479fad482ebbfaa2ac080b7feca17d5e28e4b2d15615b2.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/vnd.dsse.envelope.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
{'description': 'The MIME type of the referenced manifest',
'enum': ['application/vnd.oci.image.layer.v1.tar',
'application/vnd.oci.image.layer.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+zstd',
'application/vnd.oci.image.layer.nondistributable.v1.tar',
'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
'application/vnd.dev.cosign.simplesigning.v1+json',
'application/vnd.dsse.envelope.v1+json',
'application/tar+gzip',
'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+gzip'],
'type': 'string'}
On instance['layers'][0]['mediaType']:
'spdx+json']
2022/09/21 12:56:31 error during command execution:error processing import paths in "-": error resolving image references: writing sbom: PUT https://quay.io/v2/avinkuma/proxy-webhook-f8f95c9cea9508fe8915ae3d012d15fb/manifests/sha256-6e8f5b94738b4728ab479fad482ebbfaa2ac080b7feca17d5e28e4b2d15615b2.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/vnd.dsse.envelope.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
{'description': 'The MIME type of the referenced manifest',
'enum': ['application/vnd.oci.image.layer.v1.tar',
'application/vnd.oci.image.layer.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+zstd',
'application/vnd.oci.image.layer.nondistributable.v1.tar',
'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
'application/vnd.dev.cosign.simplesigning.v1+json',
'application/vnd.dsse.envelope.v1+json',
'application/tar+gzip',
'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
'application/vnd.oci.image.layer.v1.tar+gzip'],
'type': 'string'}
On instance['layers'][0]['mediaType']:
'spdx+json']
make: *** [Makefile:111: apply] Error 1
Adding --sbom=none
works.
I'm going to close this issue, since I'm not sure there's a lot we can do until quay.io supports these types.
@imjasonh Quay.io product manager here. We can certainly look into adding support for the missing types. What specifically does ko
expect here?
Hi @dmesser, as mentioned here https://github.com/ko-build/ko/blob/e1b4eade08febafdb9c35f11b54b699e5c04c5e5/docs/features/sboms.md?plain=1#L6 it seems quay yet not have support for SBOM for the pushed image. So pushing sbom in any format (spdx or cyclonedx) fails.
@avinal SPDX and CycloneDX themselves are known formats. The OCI type which is used to ship those SBOMs as an OCI artifact however is not standardized, it depends on the client that is used. SigStore seems to use these here: https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md#mediatypes
We are looking to enabling these shortly. Hence the question if ko
aligns with that or has separate type expectations. OCI types are basically mediaType
properties that need to conform to RFC 6838. There is no finite and standard list of distinct values though.
Referring to the implementation here: https://github.com/ko-build/ko/tree/main/internal/sbom only these two types are expected by ko
. I would let @imjasonh explain this more precisely.
Hey @dmesser 👋
ko
uses the sigstore-specified media types when pushing SBOMs. Previous versions of ko
pushed text/spdx
, but now push text/spdx+json
, and IIRC push application/vnd.cyclonedx+json
as well.
If the whole list of media types in sigstore's SBOM_SPEC.md were supported by Quay that would be great, and we could be free to choose which types+variants we support over time (e.g., maybe eventually add syft SBOMs).
Let me know if there's anything else I can help with!
@imjasonh sounds good. Yes, we are looking to enable the full list from SigStore.
FYI https://twitter.com/quayio/status/1580121221080891392
But still is doesn´t work (yet):
2023/03/05 21:05:03 Publishing quay.io/pamvdam/containers-57bcbf1d2df6010d41816c012905835d:latest
2023/03/05 21:05:04 existing blob: sha256:21219c3231f5a1cca4febbaf40f698b78e04a2d2860b65346ccd1c7db0ef2e92
2023/03/05 21:05:05 pushed blob: sha256:b339feae942b1dcc6edb5974d16f82c7c719464e6b41bf2561c1800c57291a06
Error: failed to publish images: error publishing ko://containers: writing sbom: PUT https://quay.io/v2/pamvdam/containers-57bcbf1d2df6010d41816c012905835d/manifests/sha256-368d87ad02e96d670654eddf4053c5b2139f3492776ea8f2ae5fdd06c281ada7.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' does not match '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?'
Failed validating 'pattern' in schema['properties']['layers']['items']['properties']['mediaType']:
{'description': 'The MIME type of the referenced manifest',
'pattern': '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?',
'type': 'string'}
On instance['layers'][0]['mediaType']:
'spdx+json']
2023/03/05 21:05:06 error during command execution:failed to publish images: error publishing ko://containers: writing sbom: PUT https://quay.io/v2/pamvdam/containers-57bcbf1d2df6010d41816c012905835d/manifests/sha256-368d87ad02e96d670654eddf4053c5b2139f3492776ea8f2ae5fdd06c281ada7.sbom: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'spdx+json' does not match '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?'
Failed validating 'pattern' in schema['properties']['layers']['items']['properties']['mediaType']:
{'description': 'The MIME type of the referenced manifest',
'pattern': '\\w+/[-.\\w]+(?:\\+[-.\\w]+)?',
'type': 'string'}
On instance['layers'][0]['mediaType']:
'spdx+json']
This might be fixed in ko
at head: https://github.com/ko-build/ko/issues/970#issuecomment-1456951250