node-core-utils
node-core-utils copied to clipboard
Reliablility report is inaccurate
I started investigating https://github.com/nodejs/node/issues/44805 when taking a look at https://github.com/nodejs/reliability/issues/388:
| Reason | sequential/test-watch-mode-inspect |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 9 (https://github.com/nodejs/node/pull/44787/, https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44686/, https://github.com/nodejs/node/pull/44771/, https://github.com/nodejs/node/pull/44711/) |
| Appeared | test-nearform-macos10.15-x64-2, test-nearform-macos10.15-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/46819/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/46857/ |
Example
not ok 3789 sequential/test-watch-mode-inspect
---
duration_ms: 120.67
severity: fail
exitcode: -15
stack: |-
timeout
TAP version 13
(node:37761) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
...
the first PR in the Failed PR row has two CI runs prior to this report: 46819, 46808
the first CI run reports a failure for this based on the previous osx1015 run https://ci.nodejs.org/job/node-test-commit-osx/47578/ that did not actually run in this build (due to java.net.ConnectException: Connection timed out) but Jenkins still considers the child of a previous run to be the child of this run.
this obviously happened in other PRS/CI runs as well, wich makes the report links and count very hard to rely on when investigating flaky test issues
The reports here are generated using the ncu-ci walk ... command from node-core-utils.
So this issue should be transferred, but I don't have write accees on the ncu repository so I cannot transfer it