Add new commands to support uploading and downloading images

[qweeah]

What is the version of your ORAS CLI

oras v1.1.0

What would you like to be added?

Had an offline discussion with @shizhMSFT and he suggests to add two commands:

• oras backup (alias: snapshot, pull-image) to support saving an image from remote registry to an OCI image layout tar archive.

Usage:
  oras backup --output <oci_layout_path> <reg>/<repo>[{:<tag>|@digest]}] [...]

• oras restore (alias: push-image) to support pushing an image from an OCI image layout tar archive to remote registry.

Usage:
  oras restore --output <reg>/<repo> <oci_layout_path>[{:<tag>|@digest]}]

Why is this needed for ORAS?

oras cp is not intuitive to use, e.g. user might use oras push to push images from an OCI layout and succeed with false positive (see #1353). Adding those two wrapper commands of oras cp can help avoid that.

Are you willing to submit PRs to contribute to this feature?

• [ ] Yes, I am willing to implement it.

[TerryHowe]

Why not backup and restore?

On Tue, Apr 23, 2024, 12:31 AM Billy Zha @.***> wrote:

What is the version of your ORAS CLI

oras v1.2.0 What would you like to be added?

Had an offline discussion with @shizhMSFT https://github.com/shizhMSFT and he suggests to add two commands:

• oras snapshot (alias: dump, backup, pull-image) to support saving an image from remote registry to an OCI image layout.

Usage: oras snapshot [flags] @.***} <oci_layout_path>[:]

• oras restore (alias: push-image) to support pushing an image from an OCI image layout to remote registry.

Usage: oras restore [flags] @.***} /[:]

Why is this needed for ORAS?

oras cp are not intuitive to uses, e.g. user might use oras push to push images from an OCI layout and succeed with false positive (see #1353 https://github.com/oras-project/oras/issues/1353). Adding those two wrapper commands of oras cp can help avoid that. Are you willing to submit PRs to contribute to this feature?

• Yes, I am willing to implement it.

— Reply to this email directly, view it on GitHub https://github.com/oras-project/oras/issues/1366, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAZNMMXDTQXO6V6M5ZEACLY6X52ZAVCNFSM6AAAAABGUFNS5WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TQMBVG44TCMI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[shizhMSFT]

Why not backup and restore?

Nice input. I've revised the proposal accordingly to oras backup and marked snapshot as an alias.

[sajayantony]

Could you also include reasoning why pull doesn't work here.

oras pull --to-oci-layout <oci_layout_path> <reg>/<repo>[{:<tag>|@digest]}] [...]

[sajayantony]

As per the call I'm ok if we limit to tar archive only dealing.

[shizhMSFT]

As per the call, here is an alternative without introducing new commands.

• oras backup --> oras pull --archive <tar_path> <name>{:<tag>|@<digest>}
• oras restore --> oras push --archive <tar_path> <name>[:<tag>[,<tag>][...]] <file>[:<type>] [...]

The major difference between oras backup/restore and oras pull/push is that oras backup/restore can take multiple images across registries and repositories.

Example usages are

# working with remote registry
oras pull localhost:5000/hello:v1 --archive hello.tar # backup hello:v1 to hello.tar
oras push localhost:5000/hello:v1 --archive hello.tar # restore hello:v1 from hello.tar

# working with entire repository
oras pull localhost:5000/hello --archive hello.tar # backup all images in hello to hello.tar
oras push localhost:5000/hello --archive hello.tar # restore all images in hello.tar

# working with multiple images
oras pull localhost:5000/hello:v1,foo,bar --archive hello.tar # backup v1, foo, and bar
oras push localhost:5000/hello:v1,foo --archive hello.tar     # restore v1 and foo, and ignore bar

# working with referrers
oras pull --include-referrers localhost:5000/hello:v1 --archive hello.tar
oras push --include-referrers localhost:5000/hello:v1 --archive hello.tar

# working with OCI layout. UX would be better if using `oras cp`.
oras pull --oci-layout layout.tar:v1 --archive hello.tar
oras pull --oci-layout layout-dir:v1 --archive hello.tar
oras push --oci-layout layout-dir:v1 --archive hello.tar

[qweeah]

oras restore --> oras push --archive <tar_path> [:[,][...]] [:] [...]

I think the usage is change to oras push <name>[:<tag>[,<tag>][...]] {<file>[:<type>] [...]|--archive <tar_path>}.

[TerryHowe]

Currently, pull skips successors though, right?

[qweeah]

Currently, pull skips successors though, right?

Depends on the type. subject are skipped unless --include-subject is specified. manifests in an index will be pulled.

[TerryHowe]

This issue is what I was requesting on https://github.com/oras-project/oras/issues/730

[shizhMSFT]

This issue is what I was requesting on #730

Yes, exactly. The difference is that the requirements are getting clearer after 1.5 years where more tools, including docker, are adopting OCI image layout.

[liamwh]

Is this being worked on at all? Would love to see it in ORAS!

[qweeah]

There is a little difference between the proposal and #730: cross-repository operation is not in scope here.

[liamwh]

Please forgive me @qweeah, I'm not sure I follow.

For my use case I just want to be able to pull and image and save it to disk, then be able to push it again to a different registry later. Is this being worked on and supported?

[qweeah]

Please forgive me @qweeah, I'm not sure I follow.

For my use case I just want to be able to pull and image and save it to disk, then be able to push it again to a different registry later. Is this being worked on and supported?

@liamwh Yes, it is already supported. You can use oras cp with --to-oci-layout(pull) and --from-oci-layout(push)

> # pull an image to local folder
> oras cp ghcr.io/oras-project/oras:v1.2.0 --to-oci-layout test:v1.2.0
✓ Copied  application/vnd.oci.image.config.v1+json                                                        1.56/1.56 kB 100.00%  626µs
  └─ sha256:9cd577b9700fa49ac6b2b8eca69731494a6212c7c7fca50669601e8068acc420
...
Copied [registry] ghcr.io/oras-project/oras:v1.2.0 => [oci-layout] test:v1.2.0
Digest: sha256:0087224dd0decc354b5b0689068fbbc40cd5dc3dbf65fcb3868dfbd363dc790b

An OCI image layout folder test will be created and contains the pulled image tagged v1.2.0

> tree .
.
└── test
    ├── blobs
    │   └── sha256
    │       ├── 0087224dd0decc354b5b0689068fbbc40cd5dc3dbf65fcb3868dfbd363dc790b
    ...
    ├── index.json
    ├── ingest
    └── oci-layout

You can push the image from the folder to your own registry like:

> # push an image from local folder
> oras cp --from-oci-layout test:v1.2.0 localhost:5000/oras:v1.2.0
✓ Copied  application/vnd.oci.image.config.v1+json                                                        1.57/1.57 kB 100.00%   86ms
  └─ sha256:306409c919f25a0c7900c4610a925b53f6fb32a251c0ec81ab8b4381296b0cf3
...
  └─ sha256:0087224dd0decc354b5b0689068fbbc40cd5dc3dbf65fcb3868dfbd363dc790b
Copied [oci-layout] test:v1.2.0 => [registry] localhost:5000/oras:v1.2.0
Digest: sha256:0087224dd0decc354b5b0689068fbbc40cd5dc3dbf65fcb3868dfbd363dc790b

[liamwh]

Ah ok great, thank you 🙏

Is it then correct to assume that this is also supported from the Go SDK?

[liamwh]

And is it possible to save the file as a tar instead of unpacking it?

[qweeah]

Saving(pulling) to a tar file is not supported, but reading(pushing) from a tar file is supported. You may tar the folder after pulling.

[liamwh]

Thanks for clarifying. Is it possible to save to a tar file instead of unpacking the contents to the file system?

[TerryHowe]

Is it possible to save to a tar file instead of unpacking the contents to the file system?

No

[liamwh]

Please forgive my ignorance, I'm just trying to understand: why not skip the extraction and support just downloading the tar, given that's what an OCI image actually is?

[TerryHowe]

Please forgive my ignorance, I'm just trying to understand: why not skip the extraction and support just downloading the tar, given that's what an OCI image actually is?

My understanding is that is exactly what this issue is about.

[qweeah]

why not skip the extraction and support just downloading the tar, given that's what an OCI image actually is?

@liamwh Image layer files can be tarball and oras will not extract layer files.

However, the image itself, consisting of layer tarball files and other metadata, is not a tarball. The OCI layout folder won't be compressed by oras.