lima ORAS and bootable containers - downloading images from registry

Description

There are some projects using "OCI Registry As Storage" (ORAS), to download their images.

They add a kernel to a container image, and thus making it into a virtual machine image and bootable.

One such Fedora/CentOS project is "bootc": https://docs.fedoraproject.org/en-US/bootc/

Unfortunately most images are read-only, which means that one needs to use an overlay or rpm-ostree.

The main difference for Lima, is that the actual image and digest is stored in a JSON manifest...

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 19253,
         "digest": "sha256:9be6d762b9cf0d18490eb004f29eb9bdb0ed2a73a4d89cc12bfad25790787e42",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 19289,
         "digest": "sha256:89fc46c0111a9af81db401a9514060779b413f978b4079125df8bc351bc813f1",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

The image itself is stored in a "layer", which means it needs oras rather than curl to download.

https://oras.land/docs/commands/use_oras_cli

{
	"schemaVersion": 2,
	"mediaType": "application/vnd.oci.image.manifest.v1+json",
	"artifactType": "application/vnd.unknown.artifact.v1",
	"config": {
		"mediaType": "application/vnd.oci.empty.v1+json",
		"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
		"size": 2,
		"data": "e30="
	},
	"layers": [
		{
			"mediaType": "application/x-qemu-qcow2+zstd",
			"digest": "sha256:2331217ee81b4c3a5161a50a93ecee9f1a9f3f1f145c8c019a40b2a5f9029f64",
			"size": 770370809,
			"annotations": {
				"org.opencontainers.image.title": "fedora-bootc-cloud-eln.qcow2.zst"
			}
		}
	],
	"annotations": {
		"org.opencontainers.image.created": "2024-01-31T12:57:14Z"
	}
}

To be able to separate these images from a normal URL, an oras:// scheme/prefix has been added.

oras pull ghcr.io/centos/fedora-bootc-cloud-disk:eln

Downloading 2331217ee81b fedora-bootc-cloud-eln.qcow2.zst
Downloaded  2331217ee81b fedora-bootc-cloud-eln.qcow2.zst
Pulled [registry] ghcr.io/centos/fedora-bootc-cloud-disk:eln
Digest: sha256:e9ddab1dd369f826eb0905c1fb460d683f57c410ff895d5c89245f4caaa705db

zstd -d fedora-bootc-cloud-eln.qcow2.zst

The base images are docker:// images, that needs to be converted using bootc-image-builder

FROM quay.io/fedora/fedora-bootc:40

FROM quay.io/centos-bootc/centos-bootc:stream9

REPOSITORY                          TAG       IMAGE ID       CREATED        SIZE
quay.io/centos-bootc/centos-bootc   stream9   595883768d0f   33 hours ago   1.34GB

Note: The "oras" scheme is also used by apptainer.org

One could look for quay.io as well, as a special case?

Jun 09 '24 07:06 afbjorklund

Using mounts needs a special image (with sshfs), since 9p has been disabled in the kernel.

https://github.com/vrothberg/lima-fedora-bootc

One also needs to make sure to use an image with cloud-init included, and with ssh enabled

dnf install -y openssh-server cloud-init sshfs

As a workaround, it is possible to add the fuse-sshfspackage and reboot

https://docs.fedoraproject.org/en-US/bootc/dnf/#_using_dnf_at_runtime

Jun 09 '24 07:06 afbjorklund

Note, as a special case Fedora CoreOS (and similar) make /usr/local writable with a symlink:

lrwxrwxrwx. 3 root root 15 Jan 31 13:55 /usr/local -> ../var/usrlocal

This enables the regular containerd (nerdctl-full) installation to work, even if /usr is read-only...

Alternatively, one could change the guestInstallPrefix to something else (and convince systemd)

lrwxrwxrwx.   3 root root    8 Jan 31 13:55 home -> var/home
lrwxrwxrwx.   3 root root    7 Jan 31 13:55 mnt -> var/mnt
lrwxrwxrwx.   3 root root    7 Jan 31 13:55 opt -> var/opt
lrwxrwxrwx.   3 root root   12 Jan 31 13:55 root -> var/roothome

Jun 09 '24 08:06 afbjorklund

The biggest downside is the enormous size and the hard-to-use, so in that sense it's like CoreOS et al...

https://github.com/lima-vm/lima/issues/1372

But one can use ORAS for the storage, without switching over to the rest of the "bootable containers"?

Another related topic would be to support IPFS for storing images, and download them with ipfs get

Jun 09 '24 08:06 afbjorklund

It would be possible to only support the oras: url, and not the rest of the bloatable containers.

Some other projects are using oci: instead, so should probably adopt that (or rather support both)

Beyond the fancy specifications and the helpers (like oras), it is just json:

https://tech.michaelaltfield.net/2024/09/03/container-download-curl-wget/

Oct 22 '24 14:10 afbjorklund

If you need an image that is only available as ORAS, the best is to set up a HTTP gateway.

https://gist.github.com/afbjorklund/6741932da10a1718f38c61713858f3ea

This would work similar to the IPFS_GATEWAY in curl, rewrite the URL to a local bridge...

oras://quay.io/podman/machine-os:5.4 -> http://localhost:8080/oras/quay.io/podman/machine-os:5.4

Apr 20 '25 14:04 afbjorklund