registry icon indicating copy to clipboard operation
registry copied to clipboard

Abstract content storage via trait and introduce OCI content storage

Open devigned opened this issue 2 years ago • 2 comments

This PR introduces the concept of a "ContentStore", which is an abstraction for storing package content. There are currently 2 implementations for a content store, local and OCI. However, it would be relatively simple to add another for a blob store like S3, Azure Blobs, R2, etc.

In this PR, the default file storage and serving directory is replaced with URIs that are served via the LocalContentStore. It should provide the same functionality as the current local file store / serving.

There is still some work to be done with the current OCI implementation to fill the image config with details more than just empty JSON. Furthermore, since the PR is using the image config media type, the artifactType must be specified, and I don't think that is yet available in oci-distribution lib. I will follow up on both of these details.

I'm opening this PR to start a discussion of implementation that will help to fill in the blanks for #87. Also, it's my first significant contribution to this repo, so I'd really appreciate guidance. Thank you!!

Testing out OCI content store support

Start a local registry

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Start warg-server

Tell warg-server to use the OCI v1.1 content store. It will default to connecting to a registry insecurely at localhost:5000

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ WARG_CONTENT_STORE=ociv1-1 cargo run -p warg-server -- --content-dir content

Publish a demo package

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ warg publish init default:hello
published initialization of package `default:hello`

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ warg publish release -i default:hello -v 0.1.0 demo/simple-grep-1.0.0.wasm
published version 0.1.0 of package `default:hello`

Verify the content is stored in the local registry

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ regctl repo ls localhost:5000
WARN[0000] Changing credential host for registry         host=index.docker.io new="[https://index.docker.io/v1//refresh-token"](https://index.docker.io/v1//refresh-token%22) orig="[https://index.docker.io/v1//access-token"](https://index.docker.io/v1//access-token%22)
default/hello

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ regctl manifest get localhost:5000/default/hello:0.1.0
WARN[0000] Changing credential host for registry         host=index.docker.io new="[https://index.docker.io/v1//refresh-token"](https://index.docker.io/v1//refresh-token%22) orig="[https://index.docker.io/v1//access-token"](https://index.docker.io/v1//access-token%22)
Name:        localhost:5000/default/hello:0.1.0
MediaType:   application/vnd.oci.image.manifest.v1+json
Digest:      sha256:ed8c29c6763e69f15204b3ed31e9a1ce5d6380a6070a67da2933d59525fcc07a
Total Size:  2.159MB

Config:
  Digest:    sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
  MediaType: application/vnd.oci.image.config.v1+json
  Size:      2B

Layers:

  Digest:    sha256:4470bd2867f49ef7119691d6e1cb9a4ecd46b79c394a7587dc78874c9382ca37
  MediaType: application/vnd.bytecodealliance.wasm.content.layer.v1+wasm
  Size:      2.159MB

devigned avatar Jun 29 '23 20:06 devigned

Discussed the PR with @calvinrp and @macovedj and will add an S3 compatible content store implementation to illustrate the way the client would interact with a content store which is able to send back signed URIs for fetch and upload.

devigned avatar Jul 11 '23 17:07 devigned

https://github.com/bytecodealliance/registry/discussions/160

lann avatar Jul 12 '23 14:07 lann