proposal for a multi server tests report, with MUST, SHOULD, MAY ...

[bourgeoa]

@michielbdejong @ylebre @edwardsph This is a proposal using MUST, SHOULD, MAY, as described in proposal issue #110 The example below use local repositories, located in solidos/workspaces

• In each tests repository : create a separate run file (run-against-nss.sh, run-against-css.sh, ....) for each server with a jest outputFile in a folder (example bellow -- if running the tests from solidos stack, the result folder will be in workspaces/test-suite/NSS, CSS, ESS, ... -- env values can be used to include/exclude tests

# init section
export RESULTS_PATH=.../test-suite/NSS/crud-results.json

# select section (
export INCLUDE_MAY=1            # by default 'MAY' is excluded
export SKIP_SHOULD_A12=1        # skip tests referenced 'SHOULD', 'A12'

# npm run jest "$@"
npm run jest -- --verbose --json --outputFile="$RESULTS_PATH" "$@" # verbose is optional

• in test-suite repo run a report builder using all folder/files created : example https://github.com/bourgeoa/test-suite/blob/main/test-suite-report.js
• and this is a simplistic example text report for the test-suite : https://github.com/bourgeoa/test-suite/blob/main/test-suite-report.txt (the values are real from 8/03/2021) -- default is SHOULD and test numbering default is Txxx to link to errors -- The MAY example includes a test id

[edwardsph]

When I started experimenting with this idea I had a script that aggregated the different results. I'll have a look at it tomorrow and see if it's worth you looking at but I suspect it was a minimal placeholder.