distribution-spec
distribution-spec copied to clipboard
Conformance test redesign
This is a redesign of the conformance tests with the following goals:
- Run as a command instead of a
go test. This will allow unit testing and avoid some of the Go testing wrappers. - Make the API testing more granular.
- Design the tests to run against different types of input data, including the various updates to image-spec.
- Minimize dependencies to avoid upstream breaking changes.
- Where possible, note unsupported APIs and fallback to alternatives.
This is still in draft for now, with this open PR for better visibility. Various remaining tasks include:
- [ ] Sparse index tests.
- [x] Edge cases on blob pushes (out of order, chunked push with data in the final put).
- [x] Pushing content with the wrong digest or an invalid digest algorithm.
- [ ] Verify all allowed HTTP status codes are appropriately handled.
- [ ] Cache/reuse auth tokens between HTTP requests.
- [ ] Rename conformance2 to conformance and delete the old conformance tests.
- [ ] Update the GitHub actions to test against multiple registries, and consider if/when failing 3rd party registries should block a merge to the distribution-spec.
- [ ] Generate an image in GitHub actions to be used by a conformance action.
- [ ] Update the conformance action.
- [ ] Update the oci-conformance repo to handle the new input (results will likely be summarized with two columns, API and Data, with green or gray status to show passing tests or unsupported).
- [ ] Add some unit tests.
Other wish list items:
- [ ] Add ability to run a single test.
- [ ] Add separate tests for foreign layers (URL defined) from non-distributable layers (specific media type).
Merging this will unblock #543. Closes #548. Closes #501. Closes #416.