ponyc
ponyc copied to clipboard
ponylang
Rearrange docgen for multiple backends
The doc gen code traverses the AST and generates documentation that is then processed by some external tool. At the moment this always outputs the same type of output. Ideally it should be able to output multiple different kinds of output, eg to target different external document generation tools, to generate ctags files, etc.
The doc gen code has a single entry point generate_docs() (in src\libponyc\pass\docgen.c), which is passed the AST of the whole program and a pass_opt_t struct which contains the command line arguments, etc.
Currently this code traverses the AST generating various sorted lists of types, methods, etc and generates appropriate output. The traversing and sorting should be separated from the output generation.
We could arrange it so that the traversing is done first and the resulting information is passed to the output stage. Then only the output stage would need to change to add a new output target.
However, different output targets may have different collection and sorting requirements. So it would probably be better to provide these as utility functions.
The rough code arrangement would then be:
- Determine which generation backend to use. Note that there may be more than one enabled on any given run, so we should check them all in turn.
- Appropriate backend is called, passing in the given AST.
- Backend calls whichever traversing, sorting, etc utility functions it wants.
- Backend generates output.
This reorganisation can be done with just the current output backend for now. Once this is done other backends can be added.
If we had a Pony Parser written in Pony, the doc generator could conceivably be pushed out to another separate package also written in Pony.
@jemc, you're on the right lines, but we need a lot more than just a parser to generate the docs, we need most of a compiler.
One of the nice things our docs do is tell you (via links) which type definition each type reference refers to. So for any field, parameter, etc you can click to see the relevant type. This is trickier than it sounds when you remember that you can have multiple types with the same name when using multiple packages, with use commands becoming relevant.
There are also quite a lot of things in the docs that the compiler sugars in, including create constructors and eq and ne functions. In addition to that we include functions inherited from provided traits and interfaces.
All of that lot and more is needed to make the docs useful. Getting all of that information requires doing everything up to and including a full type check, which is basically everything the compiler does other than the code generation back end.
Given that existing package documentation contains Markdown, any thoughts on how we would address that if there are multiple backends? Do we need converters? Just dump Markdown into HTML or whatever else the backend might be? Wondering if anyone had thought about that yet.
I think I'm going to need to pick this up soon.
Mkdocs is excellent for small documentation sets. It isn't good for larger ones. In order to improve the the standard library docs, I'm going to need to move them off Mkdocs. The tools that would be good for supporting StdLib would make creating a doc site for a small library much more difficult. So, sounds like multiple backends are going to be needed pretty soon.
I'm going to think about how to prioritize this against the other documentation related work I have (like writing it). If someone picks this up before me, awesome. Otherwise, I expect to be working on this within 2 to 4 weeks.
As per our conversation yesterday, one of the initial blockers for this work is getting a Pony AST parser in Pony. Then we can invoke ponyc to spit out an AST file that was processed up to an appropriate pass, then run a document generator written in Pony on the AST.