ratify-project
Should we have a repository to keep ratify images
Describe the solution you'd like Currently we have two uses cases for test images:
- images used for demonstration purpose ( quick start tests)
- images used for e2e tests
The images are currently kept in wabbitnetworks registry, lets use this issue to discuss:
- should we keep test images for e2e test, or should our e2e test produce these artifact as a test step. It is convenient to use prebuilt test image, but we will have to keep them up to date.
- what is the process to update test images? Today, not all of us has access to wabbitnetworks registry.
I feel once we have the tests on local private registry, we can just build those artifacts in each test instead of a prebuilt image since not everyone has access to the remote registry.
Re: 2 - the process would be to run the pipeline in the wabbitnetworks repository. The pipeline could be enhanced with this purpose if desired #389
Also - the access is easy to provide to anyone on the dev team. I gave everyone I knew of on the team access.
May I ask what is the value to build the new images every time we run the tests?
Also, doing this will test with only one registry, and this registry may or may not be used by the actual users of Ratify. To do a proper testing we need to test with few different registries because they support different capabilities. If I am to choose, the current test matrix should be:
| Registry | Support | Notes |
|---|---|---|
| MAR | OCI 1.1 | MAR is the only public cloud registry that currently supports OCI 1.1 (RC.1) |
| Docker Hub | Hybrid OCI 1.1 and OCI 1.0 | Docker Hub has a hybrid support for artifacts and referrers and it is good test case |
| GHCR | OCI 1.0 | This will test the legacy support |
I believe the goal here is to test the variety of options and not just one specific case.
In the future, we should add other public registries that have signatures.
To add to the above, I agree with what @dtzar wrote in #389 - we should populate withe different content if possible in those registries.