New Product proposal: IPFS Implementation feature parity checker

[SgtPooki]

Introduction

This is a work-in-progress proposal for creating a new product/tool that would benefit the IPFS ecosystem.

The idea is to create a tool that acts as a mashup of ipfs/interop & ipfs-shipyard/pinning-service-compliance where IPFS implementations (js-ipfs, iroh, kubo) can be displayed. Its intent wouldn't be to ensure interoperability but display interoperability gaps.

Goal

The goal of this new product/feature is multi-purpose:

1. A single location where all IPFS implementations and their functionality can be seen
2. Encourage/foster the creation, and performance improvements, of the different implementations
3. Support users' awareness of different implementations to simplify decision-making when deciding which implementation may be right for them
4. Encourage the adoption of certain features/functionality by an implementation via pressure from the community

Things that we shouldn't cover

• interoperability tests - ipfs/interop is a fairly large beast, and one thing I don't want this product to become is another huge maintenance burden. interoperability tests and insurance should remain in ipfs/interop, where they belong.
• infantile implementations - Implementations that do not pass a minimum bar (to be set) should not be included. We should have a minimum feature set that implementations must support to be included in the list. Without this barrier to entry, it would be too easy for this tool to become a list of incomplete and unmaintained implementations
• features that are too low-level - Displaying the support of different block encodings or other minutia could easily bog down the usefulness of this feature. The idea is not to cover every single thing an implementation can do, but instead to display which implementations support the most up-to-date and recommended things.
  • i.e. don't show nuance of dag-* support. Show that dag-cbor (or whatever is currently recommended) blocks can be added, removed, etc..

Proposal (TBD)

A lot of the work for this tool already exists elsewhere, so I don't want to duplicate that work. Instead, we should utilize existing tools to perform the functionality testing, and instead, focus this tool on rendering the resulting pass/fail for displayed features.

How we could consolidate the existing tests without duplicating work still needs to be fleshed out, but I wanted to ensure I wrote down my thoughts on this before investing too heavily.

Additional thoughts

• We could display benchmarking results to facilitate community pressure on certain implementations to speed up features & functionality that are slower than other implementations, but this could easily be its own tool.
• We may want to expand this tool to support minutia (dag-* and other microcosms of functionality) in the future, but I don't think we should start there.
• We could systematize ipfs/interop to be the foundation for this product, and make it easier to add implementations and feature tests.