UX: Improve the help doc and output for verification

[yizha1]

Summary

Currently only a digest showed as the output after notation verify successfully executed. What is this digest about: the digest of the signature, signature manifest or image manifest? see example below:

notation --cert $CERT_NAME verify $IMAGE
sha256:18adff7f255319415112345671bb41076de4a864eb792c35c20f0f6b4aa4c458

Currently the digest from the output is actually the digest of image manifest.

The help doc for notation verify is not accurate as well. Currently it is showed as below:

notation verify -h
Verifies OCI Artifacts

It is actually to verify the signature of an OCI artifact, not SBOM, VA or any other supply chain artifacts.

User Scenario

As a user, after I verify a signature of the supplied image or OCI artifact against the certificate successfully, I want to make sure the verification is performed on the correct image or OCI artifact against correct certificate.

Improvement

Here is one idea of improving the output after a successful verification.

notation --cert $CERT_NAME verify $IMAGE
The signature is verified against the certificate $CERT_NAME for the OCI artifact sha256:18adff7f255319415112345671bb41076de4a864eb792c35c20f0f6b4aa4c458 

NOTE: Maybe need to consider the case of multiple signatures against the same certificate.

cc @shizhMSFT @dtzar @FeynmanZhou @SteveLasker

[priteshbandi]

• [ ] We also need to make output script-friendly.
• [ ] Comment 1 - https://github.com/notaryproject/notation/pull/373#discussion_r995365764
• [ ] Comment 2 - https://github.com/notaryproject/notation/pull/373#discussion_r995378752
• [ ] Account for both successful and failure scenarios

[yizha1]

My proposal @priteshbandi @dtzar @iamsamirzon @binbin-li @patrickzheng200 @shizhMSFT @gokarnm @vaninrao10 , I will create PR to update spec verify.md later.

# Verify the artifact identified by digest
> notation verify localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111
Verify success for localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111


# Verify the artifact identified by tag
> notation verify localhost:5000/net-monitor:v1
Warning: Tag is used. Always use digest to identify the reference uniquely and immutably.

Resolve tag `v1` to digest `sha256:1111111111111111111111111111111111111111111111111111111111111111`
Verify success for localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111

# Verify success with --debug flag
> notation verify localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111
Use certificate: <certfile_relative_path>
Use trust policy: <policy_name>
Verify success for localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111

# Verify success with conditions with --debug flag
> notation verify localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111
Use certificate: <certfile_relative_path>
Use trust policy: <policy_name>

Error:
signature verification failure, enforcement level is enforced
${signature-reference}: ${error message}

Warning: 
signature verification failure, enforcement level is <>
${signature-reference}: ${error message}

Verify success for localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111

# Verify failure without --debug flag
> notation verify localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111
Verify failure for all the 2 signature(s) associated with localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111

## Verify failure with --debug flag
> notation verify localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111
Use certificate: <certfile_relative_path>
Use trust policy: <policy_name>

Error:
signature verification failed, enforcement level is enforced
${signature-reference}: ${error message}

Error:
signature verification failed, enforcement level is enforced
${signature-reference}: ${error message}

Verify failure for all the 2 signature(s) associated with localhost:5000/net-monitor@sha256:1111111111111111111111111111111111111111111111111111111111111111

[FeynmanZhou]

@yizha1 It looks better than before. I suggest updating Verify success to Verify succeeded and also updating Verify failure to signature verification failed.

I have a few questions:

1. enforcement level is enforced looks strange. Per spec, can we say signature verification level is enforced or verification is enforced?
2. Why do we need a flag --debug when verifying a signature? I assume this flag is only used to output the access logs.

[yizha1]

@yizha1 It looks better than before. I suggest updating Verify success to Verify succeeded and also updating Verify failure to signature verification failed.

I have a few questions:

1. enforcement level is enforced looks strange. Per spec, can we say signature verification level is enforced or verification is enforced?
2. Why do we need a flag --debug when verifying a signature? I assume this flag is only used to output the access logs.

1. enforced is one of the enforcement levels described in the spec. There are also verification levels, let me find whether we can improve it.
2. --debug is to dump the information for troubleshooting, not limited to access log. Users need to understand why signature verification failed and why verification succeeded from an overall verification point of view.

[FeynmanZhou]

@yizha1 Reply to 2, if --debug is to dump the information for troubleshooting, I think the output should include the HTTP request and header information.

[iamsamirzon]

@FeynmanZhou - The use cases for --debug are documented here. https://github.com/notaryproject/notation/issues/300

I don't think we need to display into HTTP request headers etc as those can be captured by a Wireshark TCP/IP capture tool as well.

[dtzar]

Looks good! Side related issue - it'd be really great if there was some way to parse the output from the responses. i.e. so someone who does notation verify localhost:5000/net-monitor:v1 could get back just the repo:digest for other purposes they might have.

[priteshbandi]

11/17: Create new issue for sign and verify output, specially when signing and verifying with tag.

[patrickzheng200]

Reopened because the implementation of this issue (https://github.com/notaryproject/notation/pull/450) is not merged yet.