doctests / examples

[plexus]

See an earlier draft pitch here (not all of it still matches my thinking, but linking for reference).

https://nextjournal.com/lambdaisland/pitch-kaocha-doctests

Some languages have "doctests", the ability to encode examples in docstrings, and have them verified by the test runner. This relies on agreeing on some syntax, e.g. elixer recognizes >>> at the start of a docstring line as code, and the following line(s) as the result, so it looks like what you would see in a REPL session.

e.g. in clojure you could imagine something like this

(defn my-sort 
  "Sorts a sequence.
  
  > (my-sort [4 1 3])
  ;;=> (1 3 4)
  "
  [s]
  (sort s))

Now while I think there's a lot of merit to this because it means any existing doc tools will also render these examples (which is part of the point), it's still really tedious to edit clojure inside strings. I'd much prefer to see something like this.

(defn my-sort
  {:examples '{(sort [4 3 5]) [3 4 5]
               (sort nil) []}}
  [s]
  (sort s))

In fact I've written code like this in the past, with the idea that it's already helpful just as inline documentation, and with the idea that it would be trivial to have kaocha run these as tests, I just never got around to the last part.

That said if we go down the metadata route, than I want to see a parallel effort to get this same syntax recognized in other tools, the main ones would be

• cljdoc
• codox
• cider

These are also not mutually exclusive, I can see us implementing both.

[alysbrooks]

I read an interesting proposal by Zach Oakes from a few years ago about interactive examples: https://gist.github.com/oakes/8db57ac808bf6ec144d627fd83a89da3. Adding interactivity is too much for the first version, but I think we should try to choose a syntax that will support tat use case.

[alysbrooks]

Codox actually has interactive examples that have no expected output: https://github.com/viebel/codox-klipse-theme. Unfortunately, I'm not sure we could use their format easily.

More prior art:

• https://github.com/Kobold/clj-doc-test works like the similar to
• the :test metadata key: https://clojure.org/reference/special_forms#def. Unfortunately limited to a single test. The pitch Arne linked to covers more details.
• https://github.com/arcmode/doctest
• https://github.com/liquidz/testdoc

The common syntax seems to use "=>" to indicate executed Clojure, with the return value appearing on the next line.

[alysbrooks]

This syntax might be too heavyweight, but we could also introduce a with-doctests macro:

(with-doctests
  (defn my-sort
    [s]
    (sort s))

  (sort [4 3 5]) [3 4 5]
  (sort nil) [])

[alysbrooks]

rich-comment-tests takes a slightly different approach: https://github.com/matthewdowney/rich-comment-tests. It also uses => but the example has them on the same line.

They already have clojure.test integration, so it may work with Kaocha already. Here's their example:

(ns some-test-ns
  (:require [clojure.test :refer :all]
    [com.mjdowney.rich-comment-tests.test-runner :as test-runner]))

(deftest rct-tests
  (test-runner/run-tests-in-file-tree! :dirs #{"src"}))

[alysbrooks]

Hyperfiddle also has their own version (which apparently inspired rich-comment-tests): https://github.com/hyperfiddle/rcf

[plexus]

First implementation here: https://github.com/lambdaisland/kaocha-doctest