acton icon indicating copy to clipboard operation
acton copied to clipboard

`testing` improvements

Open plajjan opened this issue 2 years ago • 0 comments

  • [x] rename unit_test_runner to test_runner (and vice versa since there is already a test_runner, which actually just runs unit_tests)
  • [x] remove testing.AssertError and replace its use with AssertionError (builtin)
  • [x] send Logger to test function
  • [x] add ActorTests and EnvTests classes + add to args in `test_runner(env, unit_tests, actor_tests, env_test)
  • [x] actually implement SyncActorTests
  • [x] actually implement AsyncActorTests
  • [x] actually implement EnvTests
  • [x] #1631
  • [x] implement __test_main discovery in compiler -> produce .test_root.c
  • [x] add builder support for .test_root.c
  • [x] depends on #1427
  • [x] add command to just list all tests
  • [x] add option to run a single test
  • [x] separate printing status from keeping results up to date etc
    • printing should be a separate actor that receives status updates
    • the current test runner should instantiate such an actor and send all the updates to it
  • [x] add support for JSON output
    • building on the previous item of separating printing
    • now becomes possible to run multiple test programs that all output
  • [x] do multiple test loops (per default) to verify stability and non-flakiness
    • like 10 loops? or maybe try to run for 0.1 seconds or something?
  • [x] support progress reporting, not just final status
    • useful for multiple iterations in test loop
  • [ ] build special logger that can capture output and display on error?
  • [ ] maybe add support for golden tests?
  • [ ] perf timing history
    • [x] record snapshot to a file on disk acton test --record
    • [ ] perf_data file should include history of perf snapshots, not just latest one, this way we can draw a graph over time
    • [x] acton test should read latest snapshot and display perf results including comparison
    • [ ] time module to have humanize() function to display time in human terms, like 123ns or 123.4ms, so pick suitable prefix and format so we get like 4 characters or something compact like that
    • [ ] 12ms +1 14ms -1ms 16ms -1ms

plajjan avatar Oct 08 '23 19:10 plajjan