nodejs
CITGM is in deep trouble
We got a regression in the latest v22.5.0 that affects at least yarn and npm: https://github.com/nodejs/node/issues/53902
It seems that citgm would have caught it, but for jobs end up green even when they have lots of failures.
From https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/
22:06:57 warn: @nearform/bubbleprof npm-install:| npm error Exit handler never called!
22:06:57 warn: @nearform/bubbleprof npm-install:| npm error This is an error with npm itself. Please report this error at:
22:06:57 warn: | npm error <https://github.com/npm/cli/issues>
22:06:57 warn: | npm error A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/bf0d267e-49e6-467a-bf5b-dcf938d40bcc/home/.npm/_logs/2024-07-16T20_05_57_634Z-debug-0.log
22:06:58 error: failure | The canary is dead:
22:07:03 error: @nearform/bubbleprof done| done - the test suite for @nearform/bubbleprof version 7.0.1 failed
22:07:03 (node:783693) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.
22:07:03 (node:783693) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.
22:08:09 warn: @nearform/doctor npm-install:| npm error Exit handler never called!
22:08:09 warn: @nearform/doctor npm-install:| npm error This is an error with npm itself. Please report this error at:
22:08:09 warn: | npm error <https://github.com/npm/cli/issues>
22:08:09 warn: | npm error A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/50a9288d-b4f4-4ba6-88c9-df6e8ba9bca3/home/.npm/_logs/2024-07-16T20_07_07_903Z-debug-0.log
22:08:10 error: failure | The canary is dead:
22:08:16 error: @nearform/doctor done| done - the test suite for @nearform/doctor version 8.1.0 failed
This is also filed as https://github.com/npm/cli/issues/7657, and https://github.com/nodejs/node/pull/53462 was pointed to me as a possible cause.
I think the cause is https://github.com/nodejs/node/pull/53627, but this issue is not about the regression itself. It's about the fact that CITGM somehow reported successful runs instead of failures.
@nodejs/build-infra I believe this is an issue with the Jenkins job. If I run node bin/citgm-all.js --includeTags binary-split, I can reproduce the regression and it correctly exits with code 1.
Possibilities:
- The job runs citgm via
eval:
# Using eval so we can use a complicated CITGM_COMMAND such as 'NODE_OPTIONS=--xzy citgm-all'
eval $CITGM_COMMAND "--nodedir=$npm_config_nodedir -v $CITGM_LOGLEVEL -x $PWD/report.xml -q $NPM_LOGLEVEL --tmpDir $temp"
- We have
Do not fail the build on empty test resultsturned on
I'll turn off Do not fail the build on empty test results.
Rerunning https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/ with Do not fail the build on empty test results disabled:
https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console
Rerunning https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/ with
Do not fail the build on empty test resultsdisabled: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console
With Do not fail the build on empty test results disabled, the CITGM run for 22.5.0 now fails due to having no test results.
That's better, thanks ! Now do we need to investigate why it "thinks" there are no results ?
https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console
13:55:12 Recording test results
13:55:12 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
which likely means smoker/report.xml wasn't generated:
we use -x to specify outputting to report.xml.
# Using eval so we can use a complicated CITGM_COMMAND such as 'NODE_OPTIONS=--xzy citgm-all'
eval $CITGM_COMMAND "--nodedir=$npm_config_nodedir -v $CITGM_LOGLEVEL -x $PWD/report.xml -q $NPM_LOGLEVEL --tmpDir $temp"
Here's the report.xml I get with Node.js 22.5.0 using the following command: node bin/citgm-all.js --includeTags binary-split -x $PWD/report.xml
<testsuite name="citgm">
<testcase name="binary-split-v1.0.5" time="12.196">
<system-out><![CDATA[ > [email protected] test
> standard --verbose && node test.js
npm error Exit handler never called!
npm error This is an error with npm itself. Please report this error at:
npm error <https://github.com/npm/cli/issues>
npm error A complete log of this run can be found in: /var/folders/9n/05y6s4ls78515vmnbd86hktm0000gn/T/9987debd-ab8e-402c-a690-08daa9c9d7de/home/.npm/_logs/2024-07-20T09_32_11_815Z-debug-0.log
sh: standard: command not found]]></system-out>
<failure message="module test suite failed">Error: The canary is dead: </failure>
</testcase>
<testcase name="clinic" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="debug" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="ember-cli" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="eslint" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="glob" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="got" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="gulp" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="jest" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="q" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="underscore" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="watchify" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="yeoman-generator" time="NaN">
<system-out/>
<skipped/>
</testcase>
</testsuite>
And with Node.js 22.5.1:
<testsuite name="citgm">
<testcase name="binary-split-v1.0.5" time="9.769">
<system-out><![CDATA[ added 267 packages in 7s
> [email protected] test
> standard --verbose && node test.js
TAP version 13
# ldjson file
ok 1 should be equal
# custom matcher
ok 2 should be equal
ok 3 should be equal
# long matcher
ok 4 should be equal
ok 5 should be equal
ok 6 should be equal
# matcher at index 0 check
ok 7 should be equal
ok 8 should be equal
ok 9 should be equal
# chunked input
ok 10 should be equal
# chunked input with long matcher
ok 11 should be equal
ok 12 should be equal
# lookbehind in multi character matcher
ok 13 should be equal
ok 14 should be equal
ok 15 should be equal
1..15
# tests 15
# pass 15
# ok]]></system-out>
</testcase>
<testcase name="clinic" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="debug" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="ember-cli" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="eslint" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="glob" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="got" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="gulp" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="jest" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="q" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="underscore" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="watchify" time="NaN">
<system-out/>
<skipped/>
</testcase>
<testcase name="yeoman-generator" time="NaN">
<system-out/>
<skipped/>
</testcase>
</testsuite>
FWIW another fast api regression made it through CITGM: https://github.com/nodejs/node/issues/54518
