Report skipped tests upon beforeAll hook failure

[juergba]

Description

related: #4233

describe("parent suite", function() {
  before(function() {
    throw new Error("this is a failure in a before hook");
  });

  it("test 1", function() { });
  it("test 2", function() { });

  describe("inner suite", function() {
    it("test 3", function() { });
  });
})

Before: a failing before hook ignores the three tests completely. They disappear and are not even summarised in the epilogue.

parent suite
  1) "before all" hook for "test 1"

0 passing (15ms)
1 failing

1) parent suite
     "before all" hook for "test 1":
   Error: this is a failure in a before hook
   [...]

After:

parent suite
  1) "before all" hook for "test 1"
  - test 1
  - test 2
  inner suite
    - test 3

0 / 3 passing (15ms)
3 skipped
1 failing

1) parent suite
     "before all" hook for "test 1":
  Error: this is a failure in a before hook
  [...]

Description of the Change

• add a new event EVENT_TEST_SKIPPED
• add a new test.state: skipped
• beforeAll hook: each skipped test is reported and summerised in the epilogue. The output is very similar to pending test cases, but in color red.
• there are no changes to the hook pattern, skipped tests will not be executed.
• adapt reporter SPEC and JSON

We now have four test states:

• passes
• pending: set by the user
  • test without callback it('some test')
  • static: it.skip() or describe.skip()
  • conditional: this.skip()
• skipped: test not run because of a failing hook
• failed: test failed or hook failed

We are aware of pending not being an optimal label. Nevertheless we will stick to it for historical reasons, it has been established for many years.

Applicable issues

ToDo:

• afterEach hooks
• reporter HTML

#1955 #1815

[coveralls]

Coverage increased (+0.08%) to 93.139% when pulling 06fc1d06bcce61f289645b1012efe2b175dfe42d on juergba/hook-failure into 38d579ac4156545c21a027bba640d49428d6bcfb on master.

[boneskull]

I'm confused about this vs. #4223

[thomasbrooks4]

@boneskull I don't know if this is actually much different then that issue described, but I forced failure at a beforeEach statement. I got no other pending or skipped test cases for the tests omitted after the failure while using Mocha 8.1.1

[abhinaba-ghosh]

Any update on this? would be really helpful for multiple e2e testing workflows.

[jerler1]

Any updates on this? Could really use this update.

[JoshuaKGoldberg]

Per https://github.com/mochajs/mocha/issues/1815#issuecomment-1902528105, marking this and #4223 as status: blocked for now. We can take a deeper look once we've gone through some smaller changes first.