HumanCompatibleAI
Embed code examples in docs
Problem
https://github.com/HumanCompatibleAI/imitation/pull/603#discussion_r1011673467
I guess if we had code examples that are embedded directly from other files this would (a) solve the issue of testing docs separately, (b) solve the pain of updating the code examples from the docs in a slow and sequential way, since (1) notebooks go out of sync with the docs, so it takes double the time to fix, and (2) such errors don't show up in e.g. typecheckers, static code analysis, etc. so it's easy to forget to update them and hard to debug the mistakes.
Can you elaborate a bit more what you had in mind?
If we just naively moved the current example code into say the examples/ directory we'd get the benefit of typecheckers and static code analysis, but we'd still have the problem of having to update both notebooks and examples. I'm not sure there's an easy way of avoiding this -- the notebooks are meant to be more full-fledged tutorials, versus minimal example code.
Agreed that having minimum viable example vs a longer tutorial is useful.
I think the key is to put the code in Python files for type checking, code analysis, automatic refactoring in IDEs, syntax highlighting, etc., then load them in. Is that viable?
I can think of one way of distilling notebooks into code examples: start in the notebook with a sample code snippet that is self-contained, along the lines: "this is what we will arrive at in this tutorial". Users can run the first cell and see a minimum viable demonstration. Then gradually build up to it with added complexity and nuance in the coming cells. It seems like a nice pedagogic narrative and also allows us to extract that cell automatically for the docs.
Extracting things from first cell of notebook sounds doable but tricky, would probably require a Sphinx extension. But I think it's easy to just embed files as code snippets in docs like https://stackoverflow.com/questions/33842973/how-to-include-external-file-as-sphinx-documentation-similar-to-a-code-block It does seem preferable to do that so we can easily type-check them. The other benefits seem slight to me. Many IDEs can do syntax higlighting of Python snippets in rst files anyway.