Runs: Failed AfterSuite() causes any scenario failed

[mirao]

Describe the bug

To Reproduce Suppose AfterSuite() in our CodeceptJS's test suite fails (simulated by I.assertEqual(1, 2);)

Expected behavior Other scenarios passed and only AfterSuite() fails

Actual behavior https://app.testomat.io/projects/spork/runs/01d25f66/report/test/21954381 All scenarios failed:

It looks like Testomat computes a result of scenario not only from scenario's result, but also from BeforeSuite() and AfterSuite(). But IMHO it makes no sense, as BeforeSuite() and AfterSuite() are executed only when test suite starts and ends.

This is how it looks in console output of test run on my side:

-- FAILURES:

  1) CoursesManage
       "after all" hook: AfterSuite for "Delete course":

      expected 1 to equal 2
      + expected - actual

      -1
      +2
      
      at chaiWrapper.assertEqual (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/[email protected]/node_modules/codeceptjs-chai/index.js:27:53)
      at Step.run (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/[email protected][email protected]/node_modules/codeceptjs/lib/step.js:123:49)
      at /home/mirao/workspace/codeceptjs/node_modules/.pnpm/[email protected][email protected]/node_modules/codeceptjs/lib/actor.js:138:23
  
  Scenario Steps:
  - I.assertEqual(1, 2) at Context.<anonymous> (./CoursesManage_test.ts:221:7)
  


  FAIL  | 15 passed, 1 failed   // 57s
Run with --verbose flag to see complete NodeJS stacktrace
[TESTOMATIO] 📊 Report Saved. Report URL: https://app.testomat.io/projects/spork/runs/01d25f66/report

Desktop (please complete the following information):

• OS: Ubuntu 22.04
• CodeceptJS 3.5.14, @testomatio/reporter 1.2.0

[bt-jagababu]

Hi @DavertMik @olexandr13 @poliarush @TetianaKhomenko We too facing this and its messing up with stats. Just commenting for better reach.

[DavertMik]

This is CodeceptJS issue, not related to our release cyecle. I'm moving it to next sprint.

[bt-jagababu]

@DavertMik @olexandr13 @AZANIR Can you please look into this issue? Its been highlighted in several management meeting as this screws up failure stats. @poliarush Could you please plan this in upcoming release ?

Thanks, Jaga

[olexandr13]

@bt-jagababu sorry for delay will investigate and try to fix shortly

[olexandr13]

@DavertMik waiting from decision from you in context of CodeceptJS

[DavertMik]

I'm working on this. I need to update lots of things in CodeceptJS to make it work

[bt-jagababu]

@DavertMik can you please check this?

[bt-jagababu]

Hi @DavertMik @poliarush @TetianaKhomenko @olexandr13 ,

Now that the Clusterize feature is introduced which everyone liked it, however this issue screws up the cluster results by sending the entire failing text as well.

Users are find this very annoying :(

Can you please provide high priority on this ASAP?

[DavertMik]

@bt-jagababu yes, releasing 3.7 was part of initial plan to fix this issue