Oras commands like attach, discover, and pull support an option to format output as JSON

[plooploops]

What is the version of your ORAS CLI

Version: 0.15.0 Go version: go1.19 Git commit: 94abaecb072d7c44bf484cb7070db2c94be2a028 Git tree state: clean

What would you like to be added?

An option to format output for commands like attach, discover, and pull as JSON.

Updated version per comments:

For oras discover

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <reference value>.

For oras pull

For oras pull, it would be helpful to get the downloaded artifactType, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras pull <reference value> -o json

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" : [
    "path/to/my/downloaded/artifact.json",
    "path/to/another/downloaded/artifact.json"
  ],
  "size": sizeInBytes
}

For oras attach

For oras attach, it would also be helpful to get the uploaded artifactDigest, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type> -o json

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" :  [
    "path/to/my/uploaded/artifact.json",
    "path/to/my/other/uploaded/artifact.json"
  ],
  "size": sizeInBytes
}

Why is this needed for ORAS?

This would be helpful for programmatically checking artifact status vs. parsing text lines (which could change depending on environment).

Are you willing to submit PRs to contribute to this feature?

• [ ] Yes, I am willing to implement it.

[sajayantony]

I would also hope that the json output could contain more scriptable values like fully qualified references so that clients don’t have to join strings.

[qweeah]

Thanks @plooploops for this feature request. Can you elaborate on the scenarios of programmatically checking artifact status for oras attach, discover and pull repectively?

Below are some minimal information needed in the JSON output in my mind: oras discover: digest, media type, size, artifact type (already supportted with -o json flag) and URL oras pull: digest, media type and size of the pulled artifact manifest oras attach: digest, media type, size and URL of the newly created artifact

[plooploops]

For oras discover

Thanks @qweeah, you're right that oras discover supports the -o json flag!

It would be helpful to include the URL as you had mentioned, I'm assuming in this case it would be for the artifact itself?

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "artifactUrl" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <artifactUrl value>.

For oras pull

The fields you had mentioned make sense, and it would also be helpful to get the downloaded artifactType, artifactUrl, and artifactFilepath too.

Suppose you make the following call:

oras pull <artifactUrl value>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "artifactUrl" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "artifactFilepath" : "path/to/my/downloaded/artifact.json",
  "size": sizeInBytes
}

For oras attach

The fields you had mentioned make sense, and it would also be helpful to get the uploaded artifactDigest, artifactUrl, and artifactFilePath.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "artifactDigest" : "sha256:someOtherHash", 
  "artifactUrl" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "artifactFilepath" : "path/to/my/uploaded/artifact.json",
  "size": sizeInBytes
}

If you think these make sense, I can pull the notes to the top post.

[qweeah]

@plooploops Thanks for the clarification. Here are some comments:

1. artifactUrl: I suggest to name it to reference since both OCI image and OCI artifact could have this field. And as you have described, this field specifies a URI from which this object MAY be downloaded, conforming to RFC 3986 with a scheme as defined in RFC 7230. Also user won't need this field in the result of oras pull since it's in the CLI argument.

2. artifactFilepath: I suggest to use files with an array type since more than one file might be processed. oras pull need this field. However oras attach doesn't because the blob file paths are specified by end users via the CLI arguments.

3. artifactDigest: This field is duplicated to digest field and only having one field digest would be enough.

/cc @FeynmanZhou for UX clarification and @shizhMSFT for planning.

[sajayantony]

Please use reference since thats has a well know definition in OCI. I’m curious why do we need to digest and artifactDigest?

[qweeah]

Please use reference since thats has a well know definition in OCI.

Good point. Updated inline.

I’m curious why do we need to digest and artifactDigest?

We don't need both. digest only would be enough.

[plooploops]

Updated version per comments:

For oras discover

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <reference value>.

For oras pull

For oras pull, it would be helpful to get the downloaded artifactType, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras pull <reference value>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" : [
    "path/to/my/downloaded/artifact.json",
    "path/to/another/downloaded/artifact.json"
  ],
  "size": sizeInBytes
}

For oras attach

For oras attaach, it would also be helpful to get the uploaded artifactDigest, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" :  [
    "path/to/my/uploaded/artifact.json",
    "path/to/my/other/uploaded/artifact.json"
  ],
  "size": sizeInBytes
}

Again, open for comments (and thank you for the input already @sajayantony and @qweeah!)

If we think this covers a good first version, then I'll pull it up to the top of the post.

[qweeah]

@plooploops This version looks good to me. One small thing to add: reference should contain scheme, e.g. http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash

[plooploops]

@qweeah Sounds good. Updated the version here (and will copy it to the top of the post).

Updated version per comments:

For oras discover

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "reference" : "http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <reference value>.

For oras pull

For oras pull, it would be helpful to get the downloaded artifactType, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras pull <reference value>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" : [
    "path/to/my/downloaded/artifact.json",
    "path/to/another/downloaded/artifact.json"
  ],
  "size": sizeInBytes
}

For oras attach

For oras attaach, it would also be helpful to get the uploaded artifactDigest, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" :  [
    "path/to/my/uploaded/artifact.json",
    "path/to/my/other/uploaded/artifact.json"
  ],
  "size": sizeInBytes
}

[sajayantony]

The reference doesn’t contain a scheme as per the spec. Also you can’t use that format in most tools and so if you pipe that to say oras tag then that wouldn’t work. My vote is to follow the spec and use fqdn/repo:tag

[plooploops]

I'm interested in simplifying @sajayantony. Which spec are you referring to in this case?

Using "<myacr>.azurecr.io/my/repo@sha256:someHash" instead would make subsequent automation easier, for example, when looking to pull in your manifest with oras manifest fetch.

[sajayantony]

I was referring to about only removing the http(s) prefix. All other fields seem reasonable. /cc @shizhMSFT for any further input.

[plooploops]

Sounds good @sajayantony.

@qweeah are you open to removing the http(s) part for the reference field?

For example, when you call oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type> you would get something like this:

{
 "digest": "sha256:someHash",
 "mediaType": "someType",
 "artifactType": "someArtifactType",
 "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
 "files" :  [
   "path/to/my/uploaded/artifact.json",
   "path/to/my/other/uploaded/artifact.json"
 ],
 "size": sizeInBytes
}

[sajayantony]

SGTM -

Does this sound like what you had in mind @plooploops

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      -o json  \
| jq .reference | xargs -n1 oras manifest fetch

1. Ensure you ask for -o json since simple text is what it defaults to
2. Use fields to pipe to jq/oras and avoid having to concat things like repo?

[qweeah]

@qweeah are you open to removing the http(s) part for the reference field?

@plooploops Sounds good if it helps automation.

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      -o json  \
| jq .reference | xargs -n1 oras manifest fetch --output <export-file-path>

@sajayantony Worth to mention that be above use case is already supported via --export-manifest (and would be safer if user is trying sign the uploaded manifest)

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      --export-manifest <export-file-path>

[sajayantony]

Ok my example may be bad. But the fact that I can use the fully qualified reference to the next command is still valuable. I don’t have to concat stuff.

[shizhMSFT]

It is bit weird to have --output for oras pull as we already have that option.

  -o, --output string                               output directory (default ".")

Generally speaking, this proposal and #497 are proposing the same thing: It is better to let the oras CLI to output some metadata so that it can be further processed.

How about --metadata-file? For example,

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      --metadata-file <metadata-file-path>

The metadata file can be further processed later. Even if there is an error in the middle of oras attach, you still can get some logs although oras is not able to produce the final metadata file.

[plooploops]

@sajayantony @qweeah - I've updated the top comment to reflect removing the scheme <http[s]> from the reference field. I also added the -o json flag as part of the example calls for oras discover, oras pull, and oras attach.

@shizhMSFT if there's an output formatting option for -o json or --output json that would let you format your output for stdout for oras discover, oras pull, and for oras attach would be helpful.

As you mentioned, oras pull does have -o json but this will put an output file somewhere, and you would need to parse the message to get the path, and then parse the output file.

oras pull <myacr>.azurecr.io/my/repo/image@sha256:somehash -o json --allow-path-traversal

Downloading differentHash C:\path\to\output\payload.json
Downloaded  differentHash C:\path\to\output\payload.json
Pulled <myacr>.azurecr.io/my/repo/image@sha256:somehash
Digest: sha256:somehash

If you can format the output that you get from stdout to json for oras discover, oras pull, and oras attach, then you can process the response without needing to parse text. For example, Azure CLI commands support multiple output formats, and having different oras commands implementing an output formatting option would be helpful from an automation standpoint.

After looking through the metadata file example from https://github.com/oras-project/oras/issues/497, it's possible to parse the metadata file to find the relevant fields like digest or reference, but is the schema of the metadata file stable? In this case, we'd still be parsing a file.

[plooploops]

@sajayantony @qweeah @shizhMSFT

Here's a hackmd to cover what's been included so far: https://hackmd.io/@plooploops/SkJn5qONs

[FeynmanZhou]

@sajayantony @qweeah @shizhMSFT

Here's a hackmd to cover what's been included so far: https://hackmd.io/@plooploops/SkJn5qONs

The proposal looks good and now I understand the scenario that you are requesting. For the tag name, there is already a flag output -o in some commands to export the output to a file. Will it be easy for users to memorize if we use just one --output flag and provide multiple options (file, JSON, other formats)?

[shizhMSFT]

The --output option means output file path or output directory in many commands like oras pull, oras blob fetch, oras manifest fetch, etc..

This proposal is to change the CLI display to the stdout. How about naming this kind of feature as --display. It takes options like text, json, and even tty, csv, yaml in the future. Let me put them in a single example,

$ oras pull example.contoso.com/myapp:v1 --output Downloads/myapp --display json
{
  "digest": "sha256:01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
  "mediaType": "application/vnd.oci.artifact.manifest.v1+json",
  "artifactType": "application/vnd.myapp",
  "reference" : "example.contoso.com/myapp@sha256:01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
  "files" : [
    "Downloads/myapp/artifact.json",
    "Downloads/myapp/app.tar"
  ],
  "size": 532
}
$ ls Downloads/myapp
artifact.json
app.tar

[plooploops]

@FeynmanZhou we could use the -o as you had mentioned for different output types like file, JSON.

However, @shizhMSFT brought up an interesting alternative in keeping the -o and --output reserved for the file path for output, and then using --display json to display the results in JSON to stdout.

In the hackmd I was using the --format json option instead, but I do like the option you had mentioned @shizhMSFT.

I updated the hackmd to reflect the proposal (and pull in your example @shizhMSFT) so we can have options for getting files, and for formatting responses to stdout as JSON.

[sajayantony]

Reviving this back up for 1.2.0. Improving the scriptability of oras is important. I think having individual issues for each of these commands would be good to discuss the respective outputs while implementing it. @FeynmanZhou @shizhMSFT