chainloop icon indicating copy to clipboard operation
chainloop copied to clipboard

feat: attestation usability improvements

Open migmartri opened this issue 1 year ago • 4 comments

I have a couple of usability suggestions that might be worth implementing

  • If we are using the discovery mode, i.e do not provide kind, and ends up falling back to artifact kind, we should explain that it didn't fit any other material kind.

  • If we pass explicitly a kind we should probably show the error message too, so instead of

chainloop att add --value ~/Downloads/jira.cyclonedx.json --kind SBOM_CYCLONEDX_JSON
ERR adding material: crafting material: invalid cyclonedx sbom file: unexpected material type

it should show

oesn't validate with http://cyclonedx.org/schema/bom-1.6.schema.json#                                
  [I#/components/0] [S#/properties/components/items/$ref] doesn't validate with '/definitions/component'                                 
    [I#/components/0/licenses] [S#/definitions/component/properties/licenses/$ref] doesn't validate with '/definitions/licenseChoice'    
      [I#/components/0/licenses] [S#/definitions/licenseChoice/oneOf] oneOf failed                                                       
        [I#/components/0/licenses/0] [S#/definitions/licenseChoice/oneOf/0/items/required] missing properties: 
  • We should be able to provide a name so we do not need to use the autogenerated one. Tracked by: https://github.com/chainloop-dev/chainloop/issues/911

wdyt?

### Tasks
- [ ] explain discovery mode resolution when falling back
- [ ] output error when it's not part of autodiscovery
- [ ] https://github.com/chainloop-dev/chainloop/issues/911
- [ ] Document attestation modes https://github.com/chainloop-dev/chainloop/issues/901
- [ ] Look into accepting invalid artifacts

migmartri avatar Jun 11 '24 07:06 migmartri

Regarding the last topic, here's the task for it with in depth explanation: https://github.com/chainloop-dev/chainloop/issues/911

javirln avatar Jun 11 '24 07:06 javirln

Thanks @migmartri. I'm afraid malformed documents are going to be a frequent issue. Given the fact that Chainloop's role is attesting, it might be reasonable to allow skipping the validation and still store it as a SBOM_CYCLONEDX_JSON, probably with an annotation validated=false. This way, Chainloop is attesting that the document exists but it's not validated so it cannot be fully trusted.

jiparis avatar Jun 11 '24 10:06 jiparis

Thanks @migmartri. I'm afraid malformed documents are going to be a frequent issue. Given the fact that Chainloop's role is attesting, it might be reasonable to allow skipping the validation and still store it as a SBOM_CYCLONEDX_JSON, probably with an annotation validated=false. This way, Chainloop is attesting that the document exists but it's not validated so it cannot be fully trusted.

Completely agree here. If a validation skip feature is implemented I think it should be explicit (user has to declare that is the desired outcome) and then that must be reflected in the metadata. My two cents.

anoncam avatar Jun 12 '24 04:06 anoncam

Thanks for the feedback. I agree with you both. What I am not sure about is if this feature should be either.

  • a) skip validation or
  • b) continue if validation error

I am leaning towards b since it can inform us whether the document was valid. If we skip the validation, we lose that info.

In other words, I'd implement something like att add --soft-validation to add the evidence regardless of the validation output. The result will contain two extra annotations

  • valid: true|false
  • soft-validation: true|false`

These two annotations would allow users for example to create a policy that enforces not to use this feature, meaning soft-validation can't be true.

Wdyt?

migmartri avatar Jun 12 '24 07:06 migmartri