enso icon indicating copy to clipboard operation
enso copied to clipboard

If `group_builder.teardown` fails (panics), the test runner fails and does not print the failed group

Open radeusgd opened this issue 1 year ago • 0 comments

Repro:

from Standard.Base import all
from Standard.Test import all

add_specs suite_builder =
    suite_builder.group "ok group" group_builder->
        group_builder.teardown <|
            IO.println "OK teardown called"

        group_builder.specify "spec 1" <|
            2.should_equal 2

        group_builder.specify "spec 2" <|
            2.should_equal 3

    suite_builder.group "failing group" group_builder->
        group_builder.teardown <|
            Panic.throw "failed to cleanup"
        group_builder.specify "my spec" <|
            IO.println "Running my spec! seconds before failure..."
            2.should_equal 2

    suite_builder.group "next group" group_builder->
        group_builder.specify "next spec" <|
            IO.println "Running next spec!"
            2.should_equal 2

main =
    suite = Test.build suite_builder->
        add_specs suite_builder
    suite.run_with_filter

Actual behaviour

OK teardown called
[FAILED] ok group: [1/2, 65ms]
    - spec 1 [26ms]
    - [FAILED] spec 2 [39ms]
        Reason: 2 did not equal 3 (at X:\NBO\repr\testing.enso:13:13-28).
Running my spec! seconds before failure...
Execution finished with an error: failed to cleanup
        at <enso> Panic.throw(Internal)
        at <enso> <anonymous><arg-1>(testing.enso:17:13-43)
        at <enso> null(Internal)
        at <enso> <anonymous>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Group.enso:37:37-40)
        at <enso> case_branch(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Helpers.enso:42:13-34)
        at <enso> Helpers.run_specs_from_group(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Helpers.enso:33-43)
        at <enso> Helpers.run_group_with_filter(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Helpers.enso:23:5-45)
        at <enso> <anonymous>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Suite.enso:85:27-73)
        at <enso> Function.<<(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Function.enso:46:26-38)
        at <enso> wrapped_function(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Internal\Array_Like_Helpers.enso:71:18-27)
        at <enso> Array_Like_Helpers.vector_from_function(Internal)
        at <enso> Array_Like_Helpers.vector_from_function(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Internal\Array_Like_Helpers.enso:86:15-68)
        at <enso> Vector.map(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Data\Vector.enso:620:9-93)
        at <enso> Vector.flat_map<arg-0>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Data\Vector.enso:658:9-37)
        at <enso> Vector.flat_map(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Base\0.0.0-dev\src\Data\Vector.enso:658:9-47)
        at <enso> Suite.run_with_filter<arg-1>(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Suite.enso:84-87)
        at <enso> Test_Reporter.wrap_junit_testsuites(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Test_Reporter.enso:22:14-19)
        at <enso> Suite.run_with_filter(X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Test\0.0.0-dev\src\Suite.enso:83-87)
        at <enso> testing.main(testing.enso:30:5-25)

Expected behaviour

Something like:

OK teardown called
[FAILED] ok group: [1/2, 65ms]
    - spec 1 [26ms]
    - [FAILED] spec 2 [39ms]
        Reason: 2 did not equal 3 (at X:\NBO\repr\testing.enso:13:13-28).
Running my spec! seconds before failure...
[CLEANUP FAILED] failing group: [1/1, 65ms]
    - my spec [26ms]
Failed to teardown the group: failed to cleanup
<stack trace>
next group: [1/1, 65ms]
    - next spec [26ms]
  1. We should still get the report for the group that failed to cleanup, as it contains valuable info about tests.
  2. As is now, we should get feedback that the group failed to cleanup, probably with a stacktrace showing the cause.
  3. The test suite should still fail, even if all other groups have succeeded - to prevent us from merging test suites with badly written cleanups that fail when they shouldn't.
  4. IMO the subsequent groups should still run. i. This may be controversial. I guess in some cases if cleanup of one group failed, this may cause other failures in subsequent groups if they are related. ii. But IMO groups should be written in an independent way so that such failures should be rare. iii. Having feedback if other groups are passing/failing is more valuable so I think they should still be run.

radeusgd avatar Feb 12 '24 19:02 radeusgd