[QCheck Alcotest] Improve the error message

[sir4ur0n]

When using QCheck with Alcotest (qcheck-alcotest), when an error occurs, the logs are rather unceremonious, and in particular, the logs show a confusing (and arguably useless) error, see e.g.:

test `My awesome QCheck test` failed on ≥ 1 cases:
My-Pretty-Printed-Value

[exception] test `My awesome QCheck test` failed on ≥ 1 cases:
My-Pretty-Printed-Value

            Raised at file "src/core/QCheck.ml", line 1726, characters 6-38
            Called from file "src/alcotest-engine/core.ml", line 277, characters 17-23
            Called from file "src/alcotest-engine/monad.ml", line 34, characters 31-35

I think there are several things that can be improved:

1. The useful part of the message is duplicated (test name, pretty printed value, number of cases, etc.)
2. The error (Raised at...) is unnecessary (and actually confusing to users, at least it was to me :smile: )
3. Some content around the pretty print of generated values (e.g. if the generated value is the empty string "", I think it is easy to not understand what is going on)

Can the situation be improved? I have no idea if this is a QCheck problem, a QCheck + Alcotest problem, or an Alcotest problem (e.g. limited Alcotest integration API).

Disclaimer: I am coming from the Haskell ecosystem where the Hedgehog library has excellent error reporting, see e.g. this blog article:

   ┏━━ test/Article.hs ━━━
44 ┃ prop_roundtrip :: Property
45 ┃ prop_roundtrip =
46 ┃   property $ do
47 ┃     x <- forAll genUser
   ┃     │ User { userId = "00000" , userName = "stephanie" }
48 ┃     tripping x Aeson.encode Aeson.eitherDecode
   ┃     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   ┃     │ ━━━ Intermediate ━━━
   ┃     │ "{\"userName\":\"stephanie\",\"userId\":\"00007\"}"
   ┃     │ ━━━ - Original) (+ Roundtrip ━━━
   ┃     │ - Right User { userId = "00000" , userName = "stephanie" }
   ┃     │ + Right User { userId = "00007" , userName = "stephanie" }

I reckon some parts may not be possible (e.g. printing the code or the line numbers) for QCheck, but this is a great example of error message: one can see the generated value (between lines 47 and 48), the compared values that failed (last 2 lines), the assertion that failed, etc.

I hope this issue is not taken as an offense, I reckon the tremendous amount of work that was put in QCheck already and I respect that :bow:

Thank you :pray:

[c-cube]

No worries. I think hedgehog has had a lot more effort put into it; template haskell and typeclasses also certainly don't hurt.

The alcotest backend was added a while ago on a request for someone else (I forget), but it's not the one I tend to personally use. Contributions here would be very welcome! Give a try maybe to the custom qcheck reporter, which also has a few options for short/long runs and verbosity control?

[sir4ur0n]

I just gave a shot to run_tests_main and the output is indeed better!

However I did not manage to get the output documented for this function :open_mouth:

Here's how my run appears (2 tests: 1 failing, 1 succeeding, I tried with and without the --long and --verbose options):

random seed: 197682039

--- Failure --------------------------------------------------------------------

Test My failing test failed (0 shrink steps):

()
================================================================================
failure (1 tests failed, 0 tests errored, ran 2 tests)

As you can see, this is quite different from https://c-cube.github.io/qcheck/0.17/qcheck-core/QCheck_base_runner/index.html#val-run_tests_main (no columns, no summary of tests, time, etc)

The code (I may have done something wrong):

let () =                                                                     
  QCheck_base_runner.run_tests_main                                          
    ~argv:[|"--verbose"; "--long"|]                                          
    [ QCheck.Test.make ~name:"My failing test" QCheck.unit (fun _ -> false); 
      QCheck.Test.make ~name:"My succeeding test" QCheck.unit (fun _ -> true)
    ]                                                                        

That being said, this runner solves points 1 and 2 of my OP, so this is a lot better already, thank you :bow:

I am not knowledgeable on Alcotest API but it seems that most "problems" occur when converting to Alcotest...

To expand on the use of Alcotest adapter: this looks like a nice selling point as it allows us to better put unit and property-based tests together. Having a common backend to gather/run/aggregate results would be a net win (the project is pretty big)

[c-cube]

I suspect that the first argument of argv is ignored (when you call the program after all, it's the executable itself!). Try to pad it, or to run it with the options from the command line?

The upside of the alcotest backend is clear, yes. I just use the ounit one instead but improving the alcotest backend is good too.

[sir4ur0n]

Indeed, the first element is ignored :grin: Thank you, now the display is as advertised :smile:

[jmid]

Note: there are two options for the verbose output: either use run_tests with the ~verbose:true option:

utop # QCheck_base_runner.run_tests;;
- : ?handler:QCheck_base_runner.handler_gen ->
    ?colors:bool ->
    ?verbose:bool ->
    ?long:bool ->
    ?debug_shrink:out_channel option ->
    ?debug_shrink_list:string list ->
    ?out:out_channel -> ?rand:Random.State.t -> QCheck.Test.t list -> int

or use run_tests_main and run the resulting executable with option -v or --verbose from the command line.