oras artifactType in index manifest and in its "manifests" descriptors

What is the version of your ORAS CLI

1.3.0-beta.2

What would you like to be added?

I've been experimenting with using the new oras manifest index create command to construct multi-platform index manifests for the new (currently in development) OCI Registry-based provider package distribution mechanism in OpenTofu.

The index-building functionality broadly worked, but in order to give better feedback to users when they provide us the wrong kind of artifact we currently require the artifactType property to be populated in two locations:

In the index manifest's top-level artifactType property, we use a type that represents a provider version as a whole, agnostic of platform.
In each of the OpenTofu-relevant descriptors in the manifests property, we expect to find a copy of the artifactType property from the manifest itself, so that we can efficiently filter any manifests that are not relevant.

(This particular situation is primarily aimed at forward compatibility with future changes to the manifest layout that we cannot anticipate today, in case a future requirement causes us to mix two different kinds of "image" artifact into the same index, with older versions of OpenTofu ignoring the newer ones they don't understand.)

For my problem 1, I'd propose adding an --artifact-type option to oras manifest index create which has similar meaning to the option of the same name on oras push, causing the given type to be written into the top-level artifactType property.

For my problem 2, I wonder if the code that builds the index manifest could copy the artifactType property from the target manifest into the corresponding descriptor, in a similar way to how it currently populates the platform property based on information from the manifest's config (or, if https://github.com/oras-project/oras/issues/1538 is implemented, from some ORAS-specific annotations in the manifest.)

Overall then, I'd like to be able to:

Construct one or more image manifests that each have our artifactType and the appropriate platform by running oras push --artifact-type=application/vnd.opentofu.provider-target --artifact-platform $os/$arch ...
Construct the image manifest by running oras manifest index create --artifact-type=application/vnd.opentofu.provider ...
Have the final index manifest include the relevant artifactType properties, so we can quickly reject/ignore irrelevant or incompatible artifacts based only on the index manifest. without fetching any image manifests:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "artifactType": "application/vnd.opentofu.provider",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.opentofu.provider-target",
      "digest": "sha256:f22b608064cba5d2bcea544ea636acca254ff3dcfcf7e39f6218b302c07c0a80",
      "size": 616,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.opentofu.provider-target",
      "digest": "sha256:9d4fa52874a3beb018d3c22683b3c49c2e2821a29c5479454958a59b5d8fee9b",
      "size": 616,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.opentofu.provider-target",
      "digest": "sha256:7e60f35171ad8a53cfe1b5e0bb771e464b748e9468e62151712947c16e5c9618",
      "size": 618,
      "platform": {
        "architecture": "amd64",
        "os": "windows"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.opentofu.provider-target",
      "digest": "sha256:e90dba706bdf891c1c9b5a83758061d64c9a4e7f7bea6b7895b6259fb27c6bd1",
      "size": 618,
      "platform": {
        "architecture": "arm64",
        "os": "windows"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.opentofu.provider-target",
      "digest": "sha256:d8635054545b76d40048e81d46a0620e49ff1c94831d864a37ab1ac3d4cefdec",
      "size": 617,
      "platform": {
        "architecture": "amd64",
        "os": "darwin"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.opentofu.provider-target",
      "digest": "sha256:7f591880d0cc947eb723dd5091ae360e7a1ba5d49e9d8478c6f6df59021c4b5c",
      "size": 617,
      "platform": {
        "architecture": "arm64",
        "os": "darwin"
      }
    }
  ]
}

Why is this needed for ORAS?

OpenTofu's provider installer would like to be able to:

Immediately reject any index manifest that doesn't have the artifact type application/vnd.opentofu.provider, to give users good feedback that they've selected the wrong artifact.
Silently ignore any manifests in the index that don't have the artifact type application/vnd.opentofu.provider-target, so that a future version of OpenTofu could potentially introduce a new artifact type to meet an as-yet-unrealized need while allowing the publication of index manifests that are backward-compatible with older versions that only understand application/vnd.opentofu.provider-target.

We could potentially meet these requirements by downloading each index manifest that has the appropriate platform and checking its artifactType, but we'd prefer to be able to deal with this using only information in the index manifest.

Are you willing to submit PRs to contribute to this feature?

[x] Yes, I am willing to implement it.

Mar 25 '25 21:03 apparentlymart

This is a typical scenario of creating an image index for multi-plaform artifact. I am happy to see @apparentlymart proposed this idea from a real use case of OpenTofu.

Adding the artifactType property to top-level index and each manifest also complies with OCI image spec v1.1. We may want to update the proposal doc to propose the changes https://github.com/oras-project/oras/blob/main/docs/proposals/multi-arch-image-mgmt.md

@apparentlymart I saw you marked that you are willing to contribute to this feature. Are you interested in driving the proposal doc update and implementation in the community? We can work together on this proposal and bring it to ORAS.

Mar 28 '25 00:03 FeynmanZhou

@apparentlymart There is another issue we raised in the OCI image spec which is kind of relevant to OpenTofu's multi-platform artifact scenario: https://github.com/opencontainers/image-spec/issues/1216

Mar 28 '25 00:03 FeynmanZhou

Sounds good to me!

Apr 02 '25 08:04 Wwwsylvia

Hi @FeynmanZhou! Sorry for the slow response.

I'm currently on vacation but I will follow up again once I'm back at work, next week. Thanks!

Apr 03 '25 15:04 apparentlymart

Hello again! I'm back now and ready to discuss how I could help with this issue.

From @FeynmanZhou's comment I understood that perhaps I could help this move forward by opening a pull request to continue the work from https://github.com/oras-project/oras/pull/1514 by making some additions to that existing proposal, as opposed to starting a new proposal. Did I understand that correctly?

Assuming that I did understand correctly, here are some additional thoughts. Of course you can ignore this all if I misunderstood the intention of the earlier discussion. 😀

The current proposal doesn't seem to directly discuss the format of the generated index manifest at all, so my initial instinct is to propose adding a new section after the existing "CLI Specs for new subcommands" section called "Index Manifest Structure" which would link to the upstream image index specification and then define how each of the properties from that specification would be populated by the oras manifest index create and oras manifest index update subcommands.

I expect that most of that definition will be straightfoward, but would in particular include the following:

The top-level artifactType is populated based on the --artifact-type option from the command line:
- For create, the artifactType property is left unpopulated when that option isn't set, or populated with the value of the option when it is set.
- For update, any existing artifactType property is unchanged when that option isn't set, or overwritten with the value of the option when it is set.
  
  (Would it make sense to allow -artifact-type="" to explicitly unset it? I'm not sure yet if the CLI options framework ORAS uses allows distinguishing between unset vs. explicitly empty string.)
The artifactType in each of the descriptors under manifests is automatically populated to match the corresponding property in the associated manifest, without any option to customize this behavior.
Specific information on how the platform object in each descriptor under manifests is populated, based on whatever is decided in https://github.com/opencontainers/image-spec/issues/1216.

(Perhaps for now the proposal would just document the current experimental behavior of ORAS CLI, while waiting for that upstream question to be resolved? It seems like so far the use-case hasn't been well understood so I assume a decision on that question will not be made very soon.)

I would also add an additional item to each of the example blocks under "CLI Specs for new subcommands" that illustrates using --artifact-type option.

Does that seem like a good approach? Thanks!

Apr 07 '25 15:04 apparentlymart

@apparentlymart I think you understood perfectly. Feel free to raise a PR to add new sections to this proposal doc based on your ideas listed above.

For this one, it is not in the scope of this issue. We could resolve it in the future until we have it aligned in the upstream OCI community.

Specific information on how the platform object in each descriptor under manifests is populated, based on whatever is decided in https://github.com/opencontainers/image-spec/issues/1216.

Apr 15 '25 21:04 FeynmanZhou

I have proposed some additions to the proposal text in https://github.com/oras-project/oras/pull/1696. Thanks!

Apr 16 '25 21:04 apparentlymart

Thank you @apparentlymart ! I will review it

Apr 17 '25 03:04 FeynmanZhou

Hello again! I'm back now and ready to discuss how I could help with this issue.

From @FeynmanZhou's comment I understood that perhaps I could help this move forward by opening a pull request to continue the work from #1514 by making some additions to that existing proposal, as opposed to starting a new proposal. Did I understand that correctly?

Assuming that I did understand correctly, here are some additional thoughts. Of course you can ignore this all if I misunderstood the intention of the earlier discussion. 😀

The current proposal doesn't seem to directly discuss the format of the generated index manifest at all, so my initial instinct is to propose adding a new section after the existing "CLI Specs for new subcommands" section called "Index Manifest Structure" which would link to the upstream image index specification and then define how each of the properties from that specification would be populated by the oras manifest index create and oras manifest index update subcommands.

I expect that most of that definition will be straightfoward, but would in particular include the following:

The top-level artifactType is populated based on the --artifact-type option from the command line:

For create, the artifactType property is left unpopulated when that option isn't set, or populated with the value of the option when it is set.

For update, any existing artifactType property is unchanged when that option isn't set, or overwritten with the value of the option when it is set. (Would it make sense to allow -artifact-type="" to explicitly unset it? I'm not sure yet if the CLI options framework ORAS uses allows distinguishing between unset vs. explicitly empty string.)

The artifactType in each of the descriptors under manifests is automatically populated to match the corresponding property in the associated manifest, without any option to customize this behavior.

Specific information on how the platform object in each descriptor under manifests is populated, based on whatever is decided in Identifying Platform-Specific OCI Artifacts opencontainers/image-spec#1216. (Perhaps for now the proposal would just document the current experimental behavior of ORAS CLI, while waiting for that upstream question to be resolved? It seems like so far the use-case hasn't been well understood so I assume a decision on that question will not be made very soon.)

I would also add an additional item to each of the example blocks under "CLI Specs for new subcommands" that illustrates using --artifact-type option.

Does that seem like a good approach? Thanks!

Thank you @apparentlymart , this looks great! I just checked your proposal PR and left some comments.

Apr 18 '25 10:04 Wwwsylvia

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

Jun 18 '25 02:06 github-actions[bot]