tasty icon indicating copy to clipboard operation
tasty copied to clipboard

Allow test duration output to be configured

Open mbg opened this issue 2 years ago • 6 comments

Ever since tasty-0.10, the durations of individual tests (above the 0.01s threshold) and the overall duration of the test suite are displayed by the consoleTestReporter ingredient. This PR adds a new option which controls which test durations are displayed:

  • --show-durations=all displays individual and overall test durations. This is the existing behaviour and remains the default option.
  • --show-durations=overall only displays the overall test duration (in the summary at the end).
  • --show-durations=tests only displays the durations of individual tests (above the 0.01s threshold).
  • --show-durations=none displays neither.

I see two applications for wanting to control this behaviour:

  • For golden tests that want to check the test output itself (e.g. when building libraries that extend tasty's functionality). Disabling the reporting of test durations leads to more stable output.
  • For teaching purposes, students often end up obsessing over test timings when they are displayed, so being able to disable them for test suites given to students may avoid some anxiety for them.

mbg avatar Apr 17 '22 21:04 mbg

FWIW an ability to control duration output would be useful for tasty-bench.

Bodigrim avatar Apr 20 '22 17:04 Bodigrim

Please rebase onto master and force-push to try and start CI.

andreasabel avatar May 07 '22 07:05 andreasabel

@andreasabel I have rebased on master, but you (or another maintainer) needs to approve the workflow.

mbg avatar May 07 '22 23:05 mbg

I don't feel like I can make a judgement on whether this feature should be included. But I observe that no tests are included in this PR; and I think that such would be needed to accept the PR.

andreasabel avatar May 08 '22 20:05 andreasabel

Sorry about the lack of tests. I had a look at the other options for the consoleTestReporter when I implemented this new option, but couldn't find any tests for those, so there wasn't a natural place for me to add tests for this PR. The contribution guidelines for this repository also don't state that tests are required, so I thought it would be ok without them.

Let me if you have any preference for where/how they should be added and I will see what I can do when I next have some time to work on this.

mbg avatar May 22 '22 20:05 mbg

Let me if you have any preference for where/how they should be added and I will see what I can do when I next have some time to work on this.

Not sure either, since I am only a steward maintainer. There are some behavioral tests in coretests/ (shell scripts e.g.), but I don't know how they work. This is something to find out.

I notice that probably also the README needs to be updated (which is the documentation AFAICS), to include the new option.

andreasabel avatar May 23 '22 07:05 andreasabel

Closing as superseded by #373. Thanks for your efforts @mbg!

Bodigrim avatar Jun 28 '23 19:06 Bodigrim