As a user I want to pull locally OCI artifacts from remote registry

[ipanova]

Feature request description

podman allows to build and push image manifest/manifest index containing OCI artifacts to the registry, however there is not command to pull them back.

Suggest potential solution

podman pull can locally pull OCI artifacts in a similar way oras pull does

Have you considered any alternatives?

it is possible to use oras to pull artifacts from the registry but it is not included in standard distributions and it is inconvenient to use different tools to perform single the workflow

Additional context

Add any other context or screenshots about the feature request here.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[ipanova]

@nalind PTAL

[rhatdan]

@mtrmac This is something skopeo can do correct? Would pulling using the sha256 work?

[mtrmac]

Where should they be pulled to?

To get a local copy, skopeo copy docker://… oci:… will work.

If the goal is something “similar to podman pull, so that an artifact pull + push works”, IIRC the new artifact feature does not store the artifacts directly in c/storage, it “just“ refers to local files on the filesystem — the c/storage metadata store is just a short-term staging area in order to push the data away. If we wanted to “pull” the artifact into the same c/storage metadata store, we would need to actually allocate local location for the files.

[rhatdan]

I think we need to have a meeting of the mind on this with Nalin and others to see where we should go with this, or just fall back to telling people to use skopeo.

[ipanova]

skopeo copy with oci transport will only re-create image directory structure and not extract the artifacts from the image. See output from oras pull

$ skopeo copy docker://pulp3-source-fedora.puffy.example.com/hello-artifact1:v5 oci:.
Getting image source signatures
Copying blob 91052b444e73 done  
Copying blob d4ac5d1cc7cc done  
Copying config ca3d163bab done  
Writing manifest to image destination
Storing signatures


$ tree .
.
├── blobs
│   └── sha256
│       ├── 2881893d9870981ebda3853d508f120ebdc733164aa1e002bf0c5c608e0ab117
│       ├── 91052b444e73f3eebdb93d1fb1506597e96c92d8de9c1e3c3f36b07a57d0a18f
│       ├── ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356
│       └── d4ac5d1cc7cc29f811d6fa6ea44e8f343b8dc47438862f7ee1130140ffc2b820
├── index.json
└── oci-layout

2 directories, 6 files


$ mkdir test2 && cd test2
$ oras pull  pulp3-source-fedora.puffy.example.com/hello-artifact1:v5
Downloading 91052b444e73 vmlinuz
Downloading d4ac5d1cc7cc initrd.img
Downloaded  d4ac5d1cc7cc initrd.img
Downloaded  91052b444e73 vmlinuz
Pulled [registry] pulp3-source-fedora.puffy.example.com/hello-artifact1:v5
Digest: sha256:2881893d9870981ebda3853d508f120ebdc733164aa1e002bf0c5c608e0ab117


$ tree
.
├── initrd.img
└── vmlinuz

0 directories, 2 files


skopeo inspect  docker://pulp3-source-fedora.puffy.example.com/hello-artifact1:v5 --raw|jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.unknown.artifact.v1",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356",
    "size": 3
  },
  "layers": [
    {
      "mediaType": "application/x-initrd-img",
      "digest": "sha256:d4ac5d1cc7cc29f811d6fa6ea44e8f343b8dc47438862f7ee1130140ffc2b820",
      "size": 1911879,
      "annotations": {
        "org.opencontainers.image.title": "initrd.img"
      }
    },
    {
      "mediaType": "application/x-vmlinuz+bz",
      "digest": "sha256:91052b444e73f3eebdb93d1fb1506597e96c92d8de9c1e3c3f36b07a57d0a18f",
      "size": 6635920,
      "annotations": {
        "org.opencontainers.image.title": "vmlinuz"
      }
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2024-04-26T13:30:22Z"
  }
}

[mtrmac]

The OCI artifact itself is really a (on-registry / on-disk) storage meta-format. What is the semantics of the config/layers depends entirely on artifactType; in particular, if there is some non-OCI-artifact “native” on-filesystem format for that object, that depends on artifactType.

It may well make sense to write a tool to natively create, and consume, 1?2?10?50? major formats.

I struggle a bit with the idea that some tool should specifically consume application/vnd.unknown.artifact.v1 in any particular way, or that org.opencontainers.image.title should be interpreted as a path name. I can, somewhat, see a cause for writing values what way, but I think specifically writing a reader that makes such assumptions is, IMHO, going too far.

Of course it’s possible for such a tool to be written, but thinking of Skopeo as a generic image/artifact transporter, I don’t see that prescribing (such a non-obvious) filesystem mapping is Skopeo’s business.

If $you want an application/vnd.something.flat-file-container with a flat-file-name annotation per layer, do so! If $you want to define a “Podman machine type” artifact with more specific semantics, even better.

But, starting with an OCI “group of blobs” abstraction, and building an application/vnd.unknown.artifact.v1 “group of named blobs” abstraction, I’m … not sure what are we doing here. It feels a bit like implementing a database inside a database to me. Sure, that another layer of indirection did add a feature, but I think at some point it’s time to actually start writing applications.

[lzap]

On our container guild gathering meeting on July 31st 2024 I presented the idea Ina and me are working on and @rhatdan mentioned that implementing podman manifest pull might be a good idea. Here is the prototype I stitched together. It uses:

• podman manifest push to push artifacts
• Python wrapper around skopeo copy to download files and create symlinks

Since we plan to start publishing Fedora and CentOS netboot artifacts on quay.io later this year, we would really appreciate a pull capability for generic files. We do not have any preference on whether this should be in podman or skopeo. Thanks!

[mtrmac]

Where should they be pulled to?

[rhatdan]

Use would specify a path, if there is more then one artifact then the path could be treated as a directory if it does not exist.

[mtrmac]

So the command could not be podman pull quay.io/foo; it would have to be something like podman pull-artifact quay.io/foo /tmp/dest? Fine. That’s a shell alias around skopeo copy, for a prototype.

Is the proposal to assume some specific MIME type / file naming convention, like the application/vnd.unknown.artifact.v1 + org.opencontainers.image.title, and to use that for file names in the destination? (I think that I must object to both using an unknown MIME type, and to the misuse of the OCI-specified annotation name, if we were to productize this particular semantics).

Or maybe we want both, “generic pull” and “pull of a named-collection-of-blobs”.

[lzap]

Just a note - we need the new command to "extract" the files from layers. What I mean is to interpret the image title annotation and then creating those files in a destination directory. Making a simple skopeo copy alias will not be useful for our use case. Thanks.