exporting rustdocs to markdown books

[sam0x17]

Right now a number of users of docify (mainly Parity/Polkadot) are interested in finding a solution to embed rust docs in a larger documentation context, preferably in the form of a markdown book.

As far as markdown books go, docify already supports processing embeds in these using docify::compile_markdown!, however I had not contemplated this idea of taking an entire rust docs (like for example the entire polkadot developer hub) and converting it to a markdown book that can then be easily embedded within whatever web page.

There are a few approaches that could accomplish something like this:

1: Visitor Pattern in Docify to Grab All Doc Comments & Produce Markdown Book

This is the most docify-like approach but is probably the hardest. This would go in tandem with #8 but would also have to include logic for processing and resolving all [`SomeItem`]-style links, which means we would also have to parse all use statements and follow them to their source which can be quite complicated in the case that it is a re-export of a re-export of a re-export, or even impossible if it is a link to an item that happens to be in a third-party / non-workspace crate. This linking problem is the key barrier here that would push me in the direction of the other approaches that leverage cargo doc / rust docs directly.

2: Customizing Rust Docs to Produce Markdown

Rust docs operates off of a simple HTML template and is fully open source. It should be possible to modify it so that instead of HTML files it produces a nested folder of markdown files that functions as a valid markdown book. This would require getting familiar with how the rust docs code base is structured and a little bit of investigation into how it works, but I am certain that it is possible. This is probably the second easiest approach and solves the linking problem since rust docs itself will already be resolving all links for us when it builds the HTML files. This will also include docs for third-party items that happen to be linked to.

3: :boom: Processing Rust Docs HTML after the Fact :boom:

Another approach that will require far less investigation is writing some rust code that traverses the HTML generated by rust docs and converts it to a directory structure of markdown files. This is probably the easiest approach and solves the linking problem since rust docs itself will already be resolving all links for us when it builds the HTML files. This will also include docs for third-party items that happen to be linked to.

4: Re-Skinning Rust Docs

As a fallback plan, it is actually quite easy to re-skin/customize rust docs and embed it in some larger web page. cargo doc allows you to specify a command-line argument to include arbitrary HTML/CSS/JavaScript on every page of the generated rust-docs, so this can include CSS+JavaScript to completely override and alter how the rust docs look and/or to hide certain items. An advantage here is the rust docs HTML is already extremely SEO optimized, so if you can get it to look good, you're essentially all set. Also, the way rust docs search works is entirely in JavaScript, so there should be a function that can be called directly to produce rust docs search results that could be called as part of a larger search system. A few hours of investigation would probably lead to being able to write a JavaScript function that simply gives you the results for a given search query.

Conclusion

Approach 3 is probably the easiest to pull off and is the one I would recommend. This is a direct rust docs => markdown conversion.

Whatever we do decide to do could be added to docify with something like docify::docs_to_markdown!("crate-name");, and could potentially be scoped to a specific crate in the workspace that we want to build docs for. For the rust doc based solutions, this would probably involve shelling out directly to cargo doc which I think is fine.