proposal: unified reference docs for user-facing w3up packages

[yusefnapora]

This issue is to scope out and discuss how we can create a "unified" collection of API reference docs for the various JS packages we maintain, especially those intended for end users.

Current state

We currently have typedoc wired up to a few repos (e.g. w3protocol, w3up-client in-progress), but there are some rough edges:

• Siloed from the rest of the docs

  • Ideally, we would render the API docs in the main website, e.g. at web3.storage/docs/reference/js.
    • consistent presentation / navigation experience
    • better SEO / discoverability
    • easier to interleave & link to concept docs & tutorials

• No cross references between packages

  • if w3up-client pulls in types from ucanto, it would be rad to be able to click through to the canonical API docs for that type.

  • e.g, in the screenshot below, I can't click through to find out what a ConnectionView is
    

  • you can sort of work around the above by adding typedoc-plugin-missing-exports, which will stuff any non-exported types it finds into a catch-all <internal> module. But it loses the original doc comments, since it's effectively repackaging the built version from node_modules instead of using the source of the original package.

    

  • what I want is for the w3up-client docs to understand that it should link out to the docs for ucanto/interface (which doesn't have typedoc set up yet, but lets pretend).

Possible solutions

rendering docs in the main website

• When I last explored this, I tried using typedoc-plugin-markdown to create .md files we could render using Nextra (our next.js docs package), but I hit issues with nextra's MDX parser choking on unsupported syntax.
  • There is a docusaurus plugin though, and since docusaurus also uses MDX it surely must be possible... perhaps digging into that plugin will reveal a magic flag to make MDX-friendly output?
  • If / when we sort out the MDX issues, we would probably want to write a little typedoc plugin to generate the side nav, etc for nextra. It should be possible to tweak the docusaurus plugin, but there are likely to be some fiddly little edge cases, because of course there will.

cross references between packages

For cross references, I think there are at least two ways to implement with typedoc that might work:

1. We could try to pull all the repos we want to be included in the docs into a shared npm workspace and use typedoc's monorepo support to generate the docs for all packages at once.
   • Essentially we'd be gluing together an "ad-hoc monorepo" using git submodules or whatever, and mucking with the dependency graph to make sure that packages depend on the versions in the local tree instead of pulling from npm, so that links between packages resolve correctly.
2. We could use typedoc's externalSymbolLinkMapping option, which lets you define the URL that each external reference should resolve to.
   • This would be a nightmare to do by hand for every package, but it should be possible to automate.

Option 1 seems like it would be super brittle and hard to maintain, assuming we could get everything lined up just so to make typedoc find all the correct dependencies in the first place.

Option 2 seems more straightforward, but we'd need to write a bit of code to generate the link mappings for each package. We'd also need to make sure the link mappings for each package get added to the typedoc config for the packages that depend on them.

One option could be for each package to spit out its own link mapping to a JSON file when we generate its docs. We could include the mapping in the package bundle, so that when we're generating the docs for a package that imports it, a script can fish the mapping out of node_modules and use it to generate the typedoc config. Typedoc plugins have access to the output filenames for each reflected type, so we should be able to generate the mapping that way.

Strawman proposal

1. We write a typedoc plugin that can

• generate markdown compatible with our MDX parser

• generate a nav.json for the side menu that works with Nextra

• generate a mapping of exported symbols to the URL for the rendered docs for that symbol

  simplified link mapping example

    {
      "@ucanto/interface": { 
        "ConnectionView": "https://web3.storage/docs/reference/js/packages/@ucanto/interface/v2.0.0/interfaces/ConnectionView/"
      }
    }
  

2. We write a script that creates a typedoc config file by looking in our node_modules for link mapping files and combining them into one big JSON object and setting it as the externalSymbolLinkMapping field.

3. Each JS lib generates its own typedoc Markdown output using our plugin and includes the link mapping it its package bundle, so it ends up in node_modules for the next package to find when generating its docs.

For the moment, lets hand-wave past the bit where we rebuild the website to include the updated docs - that's probably worth thinking about if / when we agree that this is worth doing in the first place 😄

[yusefnapora]

Some existing typedoc plugins that may be usable or good reference:

• https://github.com/nlepage/typedoc-plugin-resolve-crossmodule-references
• https://www.npmjs.com/package/typedoc-monorepo-link-types

[travis]

Essentially we'd be gluing together an "ad-hoc monorepo" using git submodules or whatever, and mucking with the dependency graph to make sure that packages depend on the versions in the local tree instead of pulling from npm, so that links between packages resolve correctly.

I thiiiink you might be able to do this pretty cleanly with pnpm workspaces? you'd have a repo with a pnpm-workspace.yaml file that pointed to the root of each of the packages in the git submodules and as long as the repos also use pnpm workspace: dependencies I think everything would just... work? I do think this would fall over if any of the repos don't use pnpm so if there are repos where we don't want to do that then this probably doesn't work.

Overall the proposal sounds smart to me! I love the goals and the implementation plan sounds right from thinking about it for a few minutes here!