validate documentation snippets with XLS command line utilities

[proppy]

to proactively detect (and prevent!) issues like https://github.com/google/xls/issues/590 it would be nice to have an automated way to validate code and commands snippets documentation.

[proppy]

one way to make the documentation testable/executable could be to use something like https://jupytext.readthedocs.io/en/latest/index.html to maintain page with commands and snippets as jupyter notebook.

[proppy]

those could then be rendered as static html document using https://nbconvert.readthedocs.io/en/latest/ or integrated directly in mkdocs using https://github.com/danielfrg/mkdocs-jupyter

[cdleary]

Had previously mentioned to @proppy -- we have a presubmit test that checks /self-contained DSLX unit tests/, [*] but (like the thing that slipped by in the OP) nothing that validates commands e.g. in an environment where tools are installed.

.. [*] https://github.com/google/xls/commit/4fb2ae14fd986c1dbcc0df8900585915855c4bbc and https://github.com/google/xls/blob/b46c71e85c22d3707e3a59fdc18832b2693a0623/xls/tests/documentation_test.py

This is in part because we don't have a standard "install" environment -- we'd need a distinct workflow from our typical "bazel test" where commands are installed and available. I suppose that unlike e.g. CMake there's not a presumed installation step / environment. I guess there's rules for making dpkgs within bazel, and then we could dpkg install within the docker environment? I think the "lighter weight and easier" the better for this kind of thing, especially because we want debugging to be easy when things have gone awry.

[proppy]

This is in part because we don't have a standard "install" environment -- we'd need a distinct workflow from our typical "bazel test" where commands are installed and available.

It seems to me that we should be able to "simulate" an "installed environment" within bazel by modifying PATH and passing shell=True to the subprocess module before invoking the various snippets as shell scripts (or notebooks cell).