distribution-spec icon indicating copy to clipboard operation
distribution-spec copied to clipboard

Conformance test redesign

Open sudo-bmitch opened this issue 1 month ago • 0 comments

This is a redesign of the conformance tests with the following goals:

  • Run as a command instead of a go test. This will allow unit testing and avoid some of the Go testing wrappers.
  • Make the API testing more granular.
  • Design the tests to run against different types of input data, including the various updates to image-spec.
  • Minimize dependencies to avoid upstream breaking changes.
  • Where possible, note unsupported APIs and fallback to alternatives.

This is still in draft for now, with this open PR for better visibility. Various remaining tasks include:

  • [ ] Sparse index tests.
  • [x] Edge cases on blob pushes (out of order, chunked push with data in the final put).
  • [x] Pushing content with the wrong digest or an invalid digest algorithm.
  • [ ] Verify all allowed HTTP status codes are appropriately handled.
  • [ ] Cache/reuse auth tokens between HTTP requests.
  • [ ] Rename conformance2 to conformance and delete the old conformance tests.
  • [ ] Update the GitHub actions to test against multiple registries, and consider if/when failing 3rd party registries should block a merge to the distribution-spec.
  • [ ] Generate an image in GitHub actions to be used by a conformance action.
  • [ ] Update the conformance action.
  • [ ] Update the oci-conformance repo to handle the new input (results will likely be summarized with two columns, API and Data, with green or gray status to show passing tests or unsupported).
  • [ ] Add some unit tests.

Other wish list items:

  • [ ] Add ability to run a single test.
  • [ ] Add separate tests for foreign layers (URL defined) from non-distributable layers (specific media type).

Merging this will unblock #543. Closes #548. Closes #501. Closes #416.

sudo-bmitch avatar Dec 04 '25 19:12 sudo-bmitch