Generate CLI docs as a Docusaurus plugin

[clemyan]

What's the problem this PR addresses?

The current method of CLI docs generation is very inefficient because it is synchronously blocking and memory-intensive. It can easily take up 40% of the time needed to start the dev server.

How did you fix it?

Created a Docusaurus plugin that generates the CLI docs. This has a number of advantages over the existing method:

• It is async and happens in parallel to other Docusaurus plugins, thus much faster in terms of total build time.
• It is written in TypeScript, so type-checked and has better DX
• It obtains the command definitions by running the TypeScript source directly instead of going through a shell, saving overhead
• It writes the result to disk via Docusaurus APIs, saving memory

Performance

# Before
Warm dev startup
  Time (mean ± σ):     68.887 s ±  1.508 s    [User: 114.795 s, System: 18.496 s]
  Range (min … max):   67.447 s … 71.328 s    5 runs

Cold dev startup
  Time (mean ± σ):     68.346 s ±  0.204 s    [User: 104.264 s, System: 15.938 s]
  Range (min … max):   68.195 s … 68.675 s    5 runs

Warm Build
  Time (mean ± σ):     110.748 s ±  0.781 s    [User: 139.832 s, System: 15.310 s]
  Range (min … max):   109.902 s … 111.528 s    5 runs

Cold Build
  Time (mean ± σ):     306.749 s ± 10.032 s    [User: 1588.512 s, System: 300.103 s]
  Range (min … max):   298.550 s … 322.576 s    5 runs
  
# After
Warm dev startup
  Time (mean ± σ):     42.410 s ±  0.546 s    [User: 60.527 s, System: 6.049 s]
  Range (min … max):   41.807 s … 43.265 s    5 runs

Cold dev startup
  Time (mean ± σ):     42.275 s ±  0.326 s    [User: 59.571 s, System: 5.824 s]
  Range (min … max):   41.760 s … 42.657 s    5 runs

Warm Build
  Time (mean ± σ):     85.861 s ±  2.109 s    [User: 95.237 s, System: 5.144 s]
  Range (min … max):   83.617 s … 88.499 s    5 runs

Cold Build
  Time (mean ± σ):     149.498 s ±  7.973 s    [User: 440.898 s, System: 15.700 s]
  Range (min … max):   140.074 s … 160.208 s    5 runs

Index pages

I have also taken the opportunity to take the @yarnpkg/cli index page and adapted it to the other binaries

⚠️ URL changes

Unfortunately, generating the CLI docs as a separate plugin makes the original URL scheme conflict with the main docs plugin, so we either have to

• move all generated CLI docs under a single path prefix (e.g. /cli)
• lose state (in particular, sidebar scroll state) when navigating between pages for different binaries

I have opted to move everything under /cli

Other Changes

The change causes a few visual changes to existing stuff:

• The previous page and next page links have been removed. Should be easy to recreate but I don't feel like they have much value
• The sidebar is now sorted in lexicographical order
• The additional binaries' examples now include the yarn invocation, and thus is properly styled
• Very minor, but the column widths in the options table have shifted very slightly

Future work

This PR is ready, but I am still experimenting with a few things that may or may not make it into the PR.

• I'm trying whether it is possible to use Docusaurus's watch mechanism to hot rebuild the pages.
• Also, as you can see, there are practically no difference in warm and cold startup time. Maybe checking mtimes can avoid some work?

Checklist

• [x] I have read the Contributing Guide.

• [x] I have set the packages that need to be released for my changes to be effective.

• [x] I will check that all automated PR checks pass before the PR gets reviewed.

[clemyan]

Got hot rebuild to work, using jiti (which is also used by Docusaurus) to re-compile the command files every time a rebuild is triggered by Docusaurus's watcher. Compilation uses esbuild by incorporating some changes from #5581.

But the deploy preview is failing now, will investigate.

[clemyan]

Looks like using esbuild to compile via jiti was the culprit. Removed that integration for now to unblock the PR.

[clemyan]

Re: esbuild + jiti. Turns out code generated by esbuild is not always reentrancy-safe (i.e. it can't handle circular requires in some cases), and jiti's implementation of interopDefault triggers the reentrancy-unsafe case. It can be made to work but performance is actually worse than just using the default babel-based compilation pipeline.

Also investigated caching to improve warm start, but that does not give a significant perf boost compared to the complexity increase. The limiting factor is probably somewhere else.