KPI Dashboard: investigate Allure

[worldomonation]

Details

Take a look at jest-allure as an option for reporting.

WooCommerce already uses Allure for their reporting: https://woocommerce.github.io/woocommerce-test-reports/daily/api/allure-report/#suites

Checklist

No response

Related

https://github.com/Automattic/wp-calypso/issues/66188

[worldomonation]

Bonus: other tools and reporters for Jest are noted here: https://github.com/jest-community/awesome-jest#reporters

[worldomonation]

After spending some time investigating this, I have made the following findings:

Jest Circus & Allure incompatibility

• Jest v27 uses jest-circus as the default test runner which is incompatible with jest-allure.
• it is possible to fall back to the jest-jasmine2 runner, but this presents some immediate problems and some future problems:
  • our current Jest environment is set up assuming Circus, and so the switch to Jasmine results in open handles being left after the test has finished execution. This can be worked around by passing --forceExit in the command line, but it does not feel "right".
  • in the future, Jest will unbundle jest-jasmime2 from the bundle, meaning it would add one more dependency.
  • use of jasmine only for calypso e2e will go against the trend of reducing toolkit variables.

Jest Circus Allure Environment

• another option is the jest circus allure environment.
• however, using this results in the browser keyword throwing an error:

    ReferenceError: browser is not defined

      12 |
      13 | 	beforeAll( async () => {
    > 14 | 		page = await browser.newPage();
         | 		^
      15 | 	} );
      16 |
      17 | 	it.each`

      at specs/infrastructure/infrastructure__ssr-enabled.ts:14:3

[worldomonation]

It looks like WooCommerce uses Jest 25 for their core API tests which still uses jest-jasmine for the default test runner and hence has no issues with Allure.

[rodelgc]

It looks like WooCommerce uses Jest 25 for their core API tests which still uses jest-jasmine for the default test runner and hence has no issues with Allure.

This is correct. We were careful not to upgrade Jest to v27 because of the same incompatibility with Allure as you mentioned. We explicitly set the testRunner to jasmine here to be extra sure that we have some kind of warning documented somewhere about the danger of upgrading to v27

https://github.com/woocommerce/woocommerce/blob/8af95fb700a54767f4b5b98f31426dc0f88bc83f/packages/js/api-core-tests/jest.config.js#L48-L54

[worldomonation]

As noted in this comment I have spent about 5 hours exploring the first option explore allure-framework/allure-js to see if it offers an out-of-the-box solution..

The scope of the task was expanded slightly to see if any Allure packages can be made to work with our current framework.

The baselines are as follows:

zaqqaz/jest-allure: does not work with our framework as it relies on Jamine test runner. Easiest to get to work with our framework however, by applying the --forceExit parameter when calling.

allure-framework/allure-js: also relies on Jasmine test runner and it was considered to fork this.

ryparker/jest-circus-allure-environment: Jest supports only one string in the testEnvironment configuration parameter, which precludes the use of this package as we have our own custom testEnvironment that we use.

azohra/allure-jest-circus: similar reason to above.

---

I had the following ideas during the time I've spent understanding some Allure-related packages:

• some packages are implemented as a reporter, of which several can be defined in the Jest configuration.
• some packages replace the entire TestEnvironment with their custom environment and this offers high degree of customization.

I attempted to implement my own custom Jest Allure reporter, but after spending some time on it I have reached the conclusion that this is infeasible:

• while Jest offers the ability to hook into events, it is not granular enough. For instance, test step failures do not have a specific event in the reporter, whereas modifying the TestEnvironment would offer the ability to hook into test_fn_failure.
• importing Allure into the reporter package is tricky.

For the next 4-5 hours, I plan to investigate whether it is possible to modify our custom TestEnvironment to support Allure.