Coverage experiment

[jmid]

I've played with computing coverage to have an idea of how much ground we cover. For now, the approach takes a copy of each tested Stdlib module and instruments that with bisect_ppx.

I do not consider this to be particularly pretty... :sweat_smile: For example, this will not report any coverage on extern-defined functions Nevertheless, I consider it good enough to obtain some basic numbers.

$ make coverage-summary 
bisect-ppx-report summary --per-file
 28.84 %   109/378    src/array/array.ml
100.00 %     6/6      src/atomic/atomic.ml
 17.28 %    42/243    src/buffer/buffer.ml
  4.29 %    34/792    src/bytes/bytescp.ml
 24.75 %    25/101    src/domain/domain.ml
 27.78 %   115/414    src/ephemeron/ephemeron.ml
 36.94 %   140/379    src/floatarray/float.ml
 28.57 %   104/364    src/hashtbl/hashtbl.ml
 12.50 %     8/64     src/io/in_channel.ml
  0.00 %     0/8      src/io/out_channel.ml
 81.48 %    22/27     src/lazy/lazy.ml
 43.40 %    23/53     src/queue/queue.ml
 41.03 %    16/39     src/semaphore/semaphore.ml
 65.38 %    17/26     src/stack/stack.ml
 43.48 %    10/23     src/sys/sys.ml
 68.57 %   168/245    src/weak/weak.ml
 26.53 %   839/3162   Project coverage

These were obtained by running the below command for the instrumented modules and computing a summary and report at the end:

dune runtest -f src/array --instrument-with bisect_ppx
....
dune runtest -f src/weak --instrument-with bisect_ppx

I'm also including the HTML in a discard-able commit (don't know if it is viewable through the GitHub UI).

Note:

• For Bigarray no coverage is reported! (4/5 tested functions defined as extern it seems)
• For Bytes the simple approach needed adjusting to avoid conflicts with the Bytes required by qcheck-core
• For Domain, instrumenting a copy crashes on my machine:

  $ dune runtest -f src/domain --instrument-with bisect_ppx
  File "src/domain/dune", line 15, characters 7-23:
  15 |  (name domain_spawntree)
              ^^^^^^^^^^^^^^^^
  Command got signal SEGV.
  File "src/domain/dune", line 6, characters 7-23:
  6 |  (name domain_joingraph)
             ^^^^^^^^^^^^^^^^
  Command got signal SEGV.
  

  Some coverage is reported though - probably up to the crash
• For Dynlink - no immediate dynlink.ml or dynlink/dynlink.ml to copy
• For Out_channel coverage is for some reason reported as 0%. I'm unsure why.
• For Thread and Threadomain - no immediate thread.ml or threads/thread.ml to copy

I'm not sure we would want this to be merged. It gives us some numbers nevertheless that we can compare to, e.g., with the multiple-ts of #112