api-doc DESPERATELY needs TLC

[rhelmot]

• [x] Fix rst syntax errors that have piled up in the docstrings
• [x] Make sure that all the modules are actually included in the docs
• [ ] Add docstrings at the top of individual modules
• [ ] Add narrative documentation into api-doc/source/*.rst
• [x] https://github.com/sphinx-doc/sphinx/issues/3408
• [x] https://github.com/sphinx-doc/sphinx/issues/1592
• [ ] https://github.com/sphinx-doc/sphinx/pull/4676
• [x] Add some form of api-doc compile/lint to CI

[rhelmot]

The closing of the two sphinx issues does not seem to have totally solved the problem they were supposed to solve, seeing as pyvex.Block.size is a link to some cfgfast member, so I opened a new PR.

[github-actions[bot]]

This issue has been marked as stale because it has no recent activity. Please comment or add the pinned tag to prevent this issue from being closed.

[github-actions[bot]]

This issue has been marked as stale because it has no recent activity. Please comment or add the pinned tag to prevent this issue from being closed.

[github-actions[bot]]

This issue has been marked as stale because it has no recent activity. Please comment or add the pinned tag to prevent this issue from being closed.

[rhelmot]

The new mission, should we choose to accept it:

Having to maintain those rst files is a gigantic pain. We should have a system to autogenerate them so we no longer have to sync angr-doc PRs with every other PR. We should migrate whatever narrative documentation is left in the rst files into docstrings or the book.

The only unanswered question afaik is what to do about the structure that is added by the hand-crafted rst files. We currently have nice headers and intuitive orderings for all the modules within angr, pyvex, etc, and we'll be losing them if we move to autogeneration. Two options that jump out at me are: 1) disregard the ordering, nobody reads the api docs top to bottom, and 2) add some sort of metadata to docstrings so we know what order and heading to put them in.

[ltfish]

disregard the ordering, nobody reads the api docs top to bottom

This.

[twizmwazin]

I recently took a stab at this and got sorta stuck on making a decision. The two best options I could find are pdochttps://github.com/mitmproxy/pdoc) and sphinx. pdoc is the magic "here's a python module with docstrings, make me some html with no extra effort required". This is what I think we want, as it keeps the docs right in python source itself and doesn't require maintaining rsts. The other option was trying to improve the sphinx setup. The key benefit here is integration with readthedocs, as basically everything else about using sphinx was a negative. Notably there's very little automatic about it. While it can read docstrings, you have to at a minimum hand-write an automodule declaration for it, and the syntax is more complicated than markdown.

With that said, RTD does have beta support for using third party doc tools, with the downside of less integration, they cite the version flyout as an example. Publishing to RTD using pdoc to build is probably the best option forward, as it should be low maintenance and should be able to be implemented quickly.

[twizmwazin]

Since api docs have been moved to project repos, I'm going to consider this complete, and then in time open new issues in the project repos regarding improving their documentation individually.