hylo

hylo copied to clipboard

We ought to run the documentation through a tool that extracts examples and tests them. I have written such a tool before and would be happy to consult on the design.

I think this will be much easier to implement once there is a published binary for the compiler. Then it would be quite straightforward to write a GitHub Actions workflow to extract code blocks from the markdown and compile them.

As a prerequisite, I think we should label examples in the markdown docs in a way that allows us to extract the ones that are compilable. Some examples may not be, and it might be awkward to make every example compilable.

It appears that the language for code blocks is parsed up to the first whitespace character. So we can add anything we want afterwards, like (compilable), to indicate that the script can extract and compile it.

    ```hylo (compilable)
    public fun main() {
      print("Hello, World!")
    }
    ```

What do you think about that approach?

It's a good idea to extract and automatically test these!

In the long run, all the samples in the markdown docs should be compilable and tested. Some may require additional surrounding code to compile, and some may be expected to produce diagnostics that should be checked. If you want to add notation in that position, it should eventually be a more expressive language than compilable-or-not. I suggest you think a little about how that more expressive language should look and then implement the minimal subset required to get some of these examples tested. It should be possible to create an SPM build tool plugin that extracts tests for these examples.

There might be a different way to approach this so that we don't need extraction. We can flip the problem around and put all the documentation examples into separate .hylo files, which can be easily tested in CI. Then embed the files into the markdown documentation as code blocks.

The problem is, this approach doesn't look very easy with Gitbook. There is this GitHub Files Gitbook integration, which can embed the files into the documentation as code blocks. However, it doesn't look like there is anyway to configure this directly in markdown files in a git repository. I might be wrong, but it looks like you need to use the web editor to use the integration, which would make it not suitable I think. I will try to confirm.

Gitbook used to have a codesnippet plugin, but it looks very old and might be unsupported now. It seems that "plugins" have been dropped and replaced with "integrations."

I know that other documentation tools support something like this. For example, Material for MkDocs, which I've used before, has support for embedding external files. I've used that feature to do exactly this—embed examples in the docs as code blocks. So if there was a desire to use something other than Gitbook in future, I think this approach would be viable.

Flipping the problem was also considered. It's either extraction or injection. Injection comes with its own problems; it becomes a lot more challenging to “just write” and there's a disincentive to injecting tiny examples. It's easier for examples to get outdated with respect to the prose. Also, you need a way to break up examples so that you don't inject a whole file at a time when you want to walk the reader though something. These ideas were all invented by Knuth many years ago; see "literate programming.”

When I wrote this book (oh, nice, google's 2nd hit is a pirated copy—feel free to refer to that), I developed a hacky but effective system for extracting and testing the examples. It was able to warn me about examples that didn't have any testing annotation saying how they should be checked. There were also tools for injecting auxilliary code into the extracted examples. In the end we were automatically testing every single piece of code in the book and it was extremely effective.

We are not attached to GitBook I think; it was just something that looked like it might be useful and I suggested we try it.