Spec Compliance Test App

[lidel]

Summary

• We need a conformance test suite that implementers and service providers can run against their service to confirm it works as expected.
• ideally this would be CLI tool that takes <access-token> and the URL with API <endpoint> and runs a set of tests
  • tests should be designed to run idempotently and reduce friction (e.g. remove all pins as the first step and confirm there are no pins as the second, and remove everything as the last step)

I'm available to review / feed edge cases is anyone wants to pick this up.

Implementation details

Create JS client library

• I believe the prerequisite here is to generate an official JS client library from the Open API YAML spec at https://ipfs.github.io/pinning-services-api-spec/
• Then publish it on NPM as @ipfs-shipyard/pinning-service-client or similar
  • Prior art (generated by someone from community): https://www.npmjs.com/package/js-pinning-service-http-client

CLI Compliance Test suite

The compliance test would be a separate package (@ipfs-shipyard/pinning-service-complicance-checks) that uses the client library to run tests and exits with code 0 if there were no hard errors:

$ npx ipfs-pinning-service-complicance-checks https://service.example.com secret-token
Checking compliance of Pinning Service API at  https://service.example.com:
  (output)
Done!

Web interface (nice to have)

Would be nice to have a static website with two inputs for <access-token> and <endpoint> and "Test" button, but this is lower priority than CLI tool (we want something that can run automatically on CI to constantly validate services we list in ipfs-webui – https://github.com/ipfs/ipfs-webui/pull/1854#issuecomment-929088269)

Test scenarios

Below are things we want to test, in order:

MVP list

• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/3
• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/4
• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/5
• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/6
• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/7
• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/8
• [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/9
• [ ] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/10
• [ ] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/11
• [ ] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/13
• [ ] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/12

[lidel]

@Gozala, as we discussed yesterday:

• This could be good opportunity to create JS client for Pinning Services API
  • Could be a thin wrapper on top of code generated with https://openapi-generator.tech/, similar to https://github.com/ipfs/go-pinning-service-http-client/pull/3
• CLI tool is more useful for now, as it can be run in CI pipelines

If someone else wants to pick this up, bash + curl is fine too.

[SgtPooki]

I've moved all of the checklist items from this issue to https://github.com/ipfs-shipyard/pinning-service-compliance/issues. We should update any of those task items in the appropriate issues going forward.