Convert documentation examples to tests

[ianmackenzie]

I started converting a bunch of the examples shown in the documentation to tests; for example see commits 8025fe78ba15395cd07a0a410c0240c8e2e11282 and be3dbb11f58d2d79fe702aa40bc033163f25537c. It would be great if someone could help converting over more of them! As nice as it would be, I don't think it's feasible to use something like stoeffel/elm-verify-examples since most tests in this package require using custom comparison functions that include a small numerical tolerance for roundoff.

The general procedure would be to find examples in a particular module, convert them to tests in the corresponding tests file, then run the tests locally using elm-test to verify that they pass.

[daphee]

Hi, I'd like to help you with the tests.

I'm new to the repo but after a few glances into various files it seems like there's lots of examples, which is great, but converting them and keeping them in sync manually seems pretty tedious. I played with elm-verify-examples a little but couldn't find a way to use a custom comparison function as well.

I opened an issue asking to change to syntax of elm-verify-examples to include a way to specify a custom comparison function.

[ianmackenzie]

Thanks, the issue looks great! I've gone and added a few comments. I agree that figuring out a way to get elm-verify-examples to work in this use case is a much better approach.

[daphee]

I have some good news!

I couldn't stop thinking about how I would extend the functionality of elm-verify-examples to cover this use case. Obviously I liked my idea of introducing a line prefixed by | containing the function to be used between the assertion and expectation part.

An example from Arc2d:

    Arc2d.endPoint exampleArc
    | Expect.point2d
    --> Point2d.fromCoordinates ( 1, 3 )

So I went on and implemented that. I am not sure whether it really is a good idea to put this there because it definitely obfuscates the examples a little bit and makes them harder to read.

That's what I wanted your opinion on and that's also the reason why I haven't submitted a PR to elm-verify-exampley yet.

I have adjusted Arc2d to use the new syntax on my branch so you can see it in action!

Maybe we should explore using RegExes in seperate configuration files but I fear that those will get quite complicated as e.g. they have to check not only for Point3d/Arc2d but also whether Maybe is used. Still this way the documentation wouldn't get cluttered.

So, in order to test it yourself:

1. Check out the branch custom-test on my fork of the elm-verify-examples repo.
2. run npm install
3. run elm-make src/VerifyExamples.elm --output bin/elm.js
4. Check out the branch verify-examples on my fork of this repo.
5. In the folder of the geometry repo run node <path_to_custom-test_checkout>/bin/cli.js to generate test/Docs/OpenSolid/Arc2dSpec.elm
6. Run elm-test. It should now pass 136 tests.

[ianmackenzie]

Cool! I'll try to take a closer look this weekend, but I'm also concerned about cluttering up the examples in the docs. A couple more possibilities:

• Embed some specially formed comments like <!-- Expect.arc2d --> in the docs above each example code block to indicate what expectation function to use, in a way that doesn't show up in the final output. This would mean that all examples in a single example code block would have to have the same result type, though (can't put comments inside the code blocks because they would get escaped and shown).
• For every doc example, call toString on both the input and output expressions, then compare those strings with a special string comparator. The comparator would split the strings into number and not-number chunks, compare the not-number chunks via string equality and compare the number chunks by converting to Float and then using a tolerant comparison. Seems a bit funky but I can't think of any cases where that wouldn't work...

[ianmackenzie]

One more 'nuclear option' possibility would be to write a custom script that converted OpenSolid doc examples into tests, with custom logic to figure out the appropriate Expect.* functions to use in each case. Conveniently, I'm actually working on a package to let you write command-line scripts in Elm so this could be a cool use case! It could hopefully be a bit simpler than elm-verify-examples since it could make a few more assumptions to exactly how examples are formatted.