modernweb-dev
`summaryReporter` doesn't print main testSuite when there are nested testSuites
Given:
src/index.spec.ts
describe('a test suite', () => {
it('should pass', () => {
expect(true).toBeTrue();
});
it('should fail', () => {
expect(true).toBeFalse();
});
describe('an inner test suite', () => {
it("should pass 2", () => {
expect(undefined).toBeUndefined();
});
it("should fail 2", () => {
expect(undefined).toBeTrue();
});
describe('another inner test suite', () => {
it("should pass 3", () => {
expect(null).toBeNull();
});
it("should fail 3", () => {
expect(null).toBeTrue();
});
});
});
});
Expected:
Actual:
Quick Analysis:
Seems to work by calling logSuite instead of logResults, in line 97
https://github.com/modernweb-dev/web/blob/d5ae228f1f030a58995ac5ac5c51df4e02612981/packages/test-runner/src/reporter/summaryReporter.ts#L94-L101
However, not sure if it's expected to have the browser name on all testSuite lines, or only the root one.
If you're testing in a single browser at a time or focusing on one at a time, it's best to include the browser name only at the root level. If you're running cross-browser tests or multiple suites for different browsers, including the browser name at each suite level may be beneficial for clarity.
Good point.
However, I did further testing and this are my findings:
{
...
browsers: [
playwrightLauncher({ product: 'chromium' }),
playwrightLauncher({ product: 'firefox' }),
playwrightLauncher({ product: 'webkit' })
],
...
}
Report is independent of testing behavior. Results are always organized in this hierarchy:
testFile (browser 1) > suite(s) > test
testFile (browser 2) > suite(s) > test
testFile (browser n) > suite(s) > test
testFile 2 (browser 1) > suite(s) > test
testFile 2 (browser 2) > suite(s) > test
testFile 2 (browser n) > suite(s) > test
testFile n (browser 1) > suite(s) > test
testFile n (browser 2) > suite(s) > test
testFile n (browser n) > suite(s) > test
Similar findings with flatten: true
PS: regardless of the browser comment, the main suite on each testFile report still needs to be fixed on the report.
Unless someone wants to do it earlier, I'll try to post a PR covering all my findings in the next weeks.
I manged to implement all changes. It works fine in regular and flatten mode.
However, there seems to be a Mocha specific behavior where all definitions are always wrapped in an "invisible suite". As stated in Mocha docs:
Top-level hooks, tests and suites are all made members of an “invisible” root suite; there is only one root suite for the entire process
This explains why summary report was implemented that way. However, is this behavior also expected from other runners (e.g. Jasmine)?
In case not, this seems to be the location where Mocha results could be unwrapped. A simple validation on root suite should be enough (no tests and only one suite child).
https://github.com/modernweb-dev/web/blob/aadcbea59e3cf929cc0ab41775b66f866d5a9dcc/packages/test-runner-mocha/src/collectTestResults.ts#L57-L69
Results when no root suite is defined:
-
Regular:
-
Flatten:
Results when root suite is defined:
-
Regular:
-
Flatten:
PS: extra indentation on failed test is associated to the usage of "aegean check mark (U+10102)". Not sure if working as expected or bug.