Review how tests are registered

[samoht]

Few users have expressed the fact that listing all the tests is error prone (as you could easily forget to add a test if you don't have error-as-warning + an empty mli to guide you).

Maybe we expose a combinator like crowbar where tests are automatically added to a global list of things to test.

[aantron]

One potential issue here is that if tests are spread out over multiple files, either (1) the files have to still be referenced from somewhere to be linked, or (2) we need to tell users to pass -linkall.

[yawaramin]

I'd like to suggest a design for adding to the 'automatic list of things to test'. In the below hypothetical code snippet I will use the test suite shown in the readme example: https://github.com/mirage/alcotest/blob/main/README.md#examples

Alcotest.suite "Utils" begin fun group ->
  group "string-case" begin fun case ->
    case "Lower case" begin fun () ->
      Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")
    end;

    case "Capitalization" begin fun () ->
      Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")
    end;
  end;

  group "string-concat" begin fun case ->
    case "String mashing" begin fun () ->
      Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])
    end;
  end;

  group "list-concat" begin fun case ->
    case ~speed:`Slow "List mashing" begin fun () ->
      Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])
    end;
  end;
end

The idea of this style is that Alcotest calls the given test suites and groups with functions which have been already set up internally with _ list ref and on each call just adds the relevant item to its corresponding list. Basically save users from having to mess around with lists, and instead raise the conceptual level Alcotest suites, groups, and cases.

[ELLIOTTCABLE]

I quite like @yawaramin's approach; it looks (and feels) a lot more like the RSpec / Jest style tests that I've been used to over the years from other languages and environments.

(In fact, I think Jest even used to provide both styles of API, with the describe/it(…)-style API being optional sugar? I think that'd be a great idea here as well.)

[yawaramin]

Hey, yeah, I had something like Ruby on my mind for this. To make it easy for people to compare, pretending for a second that the module was written in Ruby, here's part of it translated to RSpec:

RSpec.describe Utils, "Utils" do
  context "string-case" do
    it "Lower case" do
      expect(To_test.lowercase("hEllO!")).to eq "hello!"
    end

    it "Capitalization" do
      expect(To_test.capitalize("world.")).to eq "World."
    end
  end
end

[yawaramin]

Alternative design:

let () =
  let open Alcotest in
  run "Utils" [
    group "string-case" [
      case "Lower case" begin fun () ->
        check string "same string" "hello!" (To_test.lowercase "hELLO!")
      end;

      case "Capitalization" begin fun () ->
        check string "same string" "World." (To_test.capitalize "world.")
      end;
    ];

    group "string-concat" [
      case "String mashing" begin fun () ->
        check string "same string" "foobar" (To_test.str_concat ["foo"; "bar"])
      end;
    ];

    group "list-concat" [
      case ~speed:`Slow "List mashing" begin fun () ->
        check (list int) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])
      end;
    ];
  ]

The idea is that instead of messing around with tuples we provide constructors of the correct types. Then we say that an Alcotest suite contains a list of groups, and a group contains a list of cases.

[dmmulroy]

Any updates on this and #393?