enso
enso copied to clipboard
If `group_builder.teardown` fails (panics), the test runner fails and does not print the failed group
Repro:
from Standard.Base import all
from Standard.Test import all
add_specs suite_builder =
suite_builder.group "ok group" group_builder->
group_builder.teardown <|
IO.println "OK teardown called"
group_builder.specify "spec 1" <|
2.should_equal 2
group_builder.specify "spec 2" <|
2.should_equal 3
suite_builder.group "failing group" group_builder->
group_builder.teardown <|
Panic.throw "failed to cleanup"
group_builder.specify "my spec" <|
IO.println "Running my spec! seconds before failure..."
2.should_equal 2
suite_builder.group "next group" group_builder->
group_builder.specify "next spec" <|
IO.println "Running next spec!"
2.should_equal 2
main =
suite = Test.build suite_builder->
add_specs suite_builder
suite.run_with_filter
Actual behaviour
OK teardown called
[FAILED] ok group: [1/2, 65ms]
- spec 1 [26ms]
- [FAILED] spec 2 [39ms]
Reason: 2 did not equal 3 (at X:\NBO\repr\testing.enso:13:13-28).
Running my spec! seconds before failure...
Execution finished with an error: failed to cleanup
at <enso> Panic.throw(Internal)
at <enso> <anonymous><arg-1>(testing.enso:17:13-43)
at <enso> null(Internal)
at <enso> <anonymous>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Group.enso:37:37-40)
at <enso> case_branch(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Helpers.enso:42:13-34)
at <enso> Helpers.run_specs_from_group(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Helpers.enso:33-43)
at <enso> Helpers.run_group_with_filter(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Helpers.enso:23:5-45)
at <enso> <anonymous>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Suite.enso:85:27-73)
at <enso> Function.<<(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Function.enso:46:26-38)
at <enso> wrapped_function(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Internal\Array_Like_Helpers.enso:71:18-27)
at <enso> Array_Like_Helpers.vector_from_function(Internal)
at <enso> Array_Like_Helpers.vector_from_function(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Internal\Array_Like_Helpers.enso:86:15-68)
at <enso> Vector.map(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Data\Vector.enso:620:9-93)
at <enso> Vector.flat_map<arg-0>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Data\Vector.enso:658:9-37)
at <enso> Vector.flat_map(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Data\Vector.enso:658:9-47)
at <enso> Suite.run_with_filter<arg-1>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Suite.enso:84-87)
at <enso> Test_Reporter.wrap_junit_testsuites(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Test_Reporter.enso:22:14-19)
at <enso> Suite.run_with_filter(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Suite.enso:83-87)
at <enso> testing.main(testing.enso:30:5-25)
Expected behaviour
Something like:
OK teardown called
[FAILED] ok group: [1/2, 65ms]
- spec 1 [26ms]
- [FAILED] spec 2 [39ms]
Reason: 2 did not equal 3 (at X:\NBO\repr\testing.enso:13:13-28).
Running my spec! seconds before failure...
[CLEANUP FAILED] failing group: [1/1, 65ms]
- my spec [26ms]
Failed to teardown the group: failed to cleanup
<stack trace>
next group: [1/1, 65ms]
- next spec [26ms]
- We should still get the report for the group that failed to cleanup, as it contains valuable info about tests.
- As is now, we should get feedback that the group failed to cleanup, probably with a stacktrace showing the cause.
- The test suite should still fail, even if all other groups have succeeded - to prevent us from merging test suites with badly written cleanups that fail when they shouldn't.
- IMO the subsequent groups should still run. i. This may be controversial. I guess in some cases if cleanup of one group failed, this may cause other failures in subsequent groups if they are related. ii. But IMO groups should be written in an independent way so that such failures should be rare. iii. Having feedback if other groups are passing/failing is more valuable so I think they should still be run.