Rendering/Format Proposal: migrate ZIPs to a single `mdbook` instance.

[nathan-at-least]

Problem

The rendering of ZIPs is bespoke, error prone, and messy. Here's my assessment of the current setup:

Pros:

• The formats and processes related to rendering are established precedent, understood (by ZIP editors at least), and working.
• The rendering process and pipeline is customized to suit the current needs.
• Multiple formats are supported, e.g. rst, md giving authors more flexibility.

Cons:

• The rendering process is a bit messy to set up initially.
• The process is hard to maintain. For example new lints or rendering features are implemented in a variety of make, shell scripts, python scripts. Understanding the dependencies, sequencing, and formats of these is tedious.
• Because the rendering is bespoke, there are not "off-the-shelf" tools we could use if we had a well-used toolchain.
• ~~The rendering output is tracked by revision control, so every commit includes a bunch of automation-generated noise (or else it fails to do so, leading to inconsistent revision history conventions). (Note: This can be addressed separately as in #731 without this proposal. However, that is yet another case where the bespoke system maintenance remains a burden.)~~ Fixed in #711.

Proposed Solution

• Migrate all ZIPs into a single mdbook instance.
• Keep the protocol spec as is (latex toolchain).
• Simplify the build process as much as possible to two steps: mdbook build and make --directory protocol.
• Integrate those two build steps into a single command (maybe mdbook build if possible or make; I prefer the former because it's too easy to begin adding complexity to a Makefile which undermines the goal of simplifying the toolchain and build process).
• Remove as many build dependencies as possible outside of mdbook dependencies managed by cargo and latex dependencies.
• ~~Remove rendering output from revision control and replace with appropriate automation, for example this recipe or other similar recipes for mdbook → github pages. (Related: #731)~~ Already done in #711.
• As much as possible encode the ZIP-specific conventions into mdbook preprocessors. For example ZIP metadata parsing/rendering, or summary table generation. (These may be developed, maintained, and packaged as a separate repo to keep this repo content-specific, or maybe it's easier to have a monolithic repo initially. I tend to prefer the former.)

My evaluation of pros/cons here:

Pros:

• A single format and rendering toolchain for ZIPs simplifies complexity for ZIP authors. If they need format/rendering help, they can look up third-party mdbook docs rather than needing to learn bespoke details from this system.
• ZIPs can now crosslink to each other within the existing mdbook framework.
• The rendered result will have a widely-used UX, such as the navigation system of the default theme, so it may be more familiar to more users. Also UX improvements from the widely used toolchain will propagate to ZIPs.
• A well-used toolchain is easy to extend with third party lints, preprocessors, and generators, e.g. link checking, theming, rendering extensions (math, diagrams, etc…).
• ZIP-specific rendering customizations have a greater potential to be useful to other projects, increasing the chances of wider support/maintenance. (e.g. maybe ZIP metadata parsing/tabulation becomes useful for other projects).
• It is easier to recreate the rendering elsewhere, such as if third party sites want to render ZIPs, or other projects want to reify the ZIP-rendering framework. For example, maybe Zcash will evolve distinct standards categories to distinguish consensus protocol vs front-end / app standards vs auxillary protocols, and those could each use a similar rendering/format framework with different editorial processes or organizations.

Cons:

• This requires migrating all rst ZIP files into mdbook format as a one-time flag-day event.
• The current system works well enough, so if it aint broke don't fix it.

Alternative Solutions

I just chose mdbook because I'm familiar with it and it's my go-to revision-control-friendly doc rendering system. Maybe there are good arguments for other frameworks with a wide userbase (maybe an rst-based one?)