Tests runs only ends in timeout despite `npx start-test-run --finish` command responds ok

[florianlefeuvre]

Describe the bug At work we have a Cypress stack using Gherkin and your reporter, and our tests are launched in parallel through cypress-parallel dependency.

Everything works fine until we try to finish the runs through npx start-test-run --finish. This command seems to work as intended in the logs (it correctly responds Run <run-id> was finished), but on Testomat.io the run doesn't finish (until the 30 minutes timeout setup in our project settings).

To Reproduce We have a .gitlab-ci.yml with a test:cypress job containing :

variables :
    TESTOMATIO: ${TESTOMATIO_TOKEN}
    TESTOMATIO_TITLE: 'Report ${env} at ${CI_PIPELINE_CREATED_AT} (pipeline: #${CI_PIPELINE_ID})'
    TESTOMATIO_RUN: ${TESTOMATIO_RUN_ID}
    TESTOMATIO_PROCEED: 1
    TESTOMATIO_ENV: ${env}
    SEVERITY_LEVEL: ''
before_script:
    - export TESTOMATIO_RUN=$(TESTOMATIO=${TESTOMATIO_TOKEN} npx start-test-run --launch | tail -1)
    - echo $TESTOMATIO_RUN > RUN_ID
script:
    - npm run parallel:$env$SEVERITY_LEVEL -- --threads ${THREADS_NUMBER}
after_script:
    - export TESTOMATIO_RUN=$(cat RUN_ID)
    - TESTOMATIO=${TESTOMATIO_TOKEN} npx start-test-run --finish

Exemple of script that can be launched through package.json :

    "run:preprod:critical": "cypress run --browser chrome --headless --env ENV=preprod,TAGS=\"not @ignore and not @exclude-from-preprod and @critical\"",
    "parallel:preprod:critical": "cypress-parallel --script run:preprod:critical --threads 4 --specsDir \"cypress/<path-to-features>/**/*.feature\"",

Expected behavior Tests runs are launched in parallel (using different threads). When the job finishes the after_script launches npx start-test-run --finish, then the tests run on testomat.io should be flagged as finished. Instead, test run is still pending and finishes with a 30 minutes timeout.

Screenshots

• Logs from CI (job finishes) :
• Job status from Gitlab (finished 7 minutes ago)
• In testomat.io, test run is still running but all tests are finished and correctly uploaded (finish state is not triggered)

Additional context We didn't have this issue before. It appeared weeks ago (despite not having updated anything in our stack). We then did try to update the reporter dependency (to 1.4.6) but it didn't help.

[DavertMik]

@florianlefeuvre

As a first guess I can assume TESTOMATIO_PROCEED=1 is used after_script so it completely blocks finishing run as it specifies that run should be kept

Move this variable to script so it would be used only on that section.

If this doesn't help please add the following variable

TESTOMATIO_DISABLE_BATCH_UPLOAD=1

This disables async processing of jobs which can also cause the similar issues

In the next Testomat.io update we will handle such cases so TESTOMATIO_DISABLE_BATCH_UPLOAD won't be needed

[DavertMik]

I re-checked this issue. Latest reporter 1.5.0 should handle this case correctly.

https://www.npmjs.com/package/@testomatio/reporter/v/1.5.0

[MathieuJouis]

Hello @DavertMik,

After upgrading to latest reporter v1.5.0, we see no change in the behavior.

The run is clearly indicated as finished on CI :

 🤩 Testomat.io Reporter v1.5.0
Finishing Run on Testomat.io...
[TESTOMATIO] Pipes: Testomatio Reporter
[TESTOMATIO] Testomatio Reporter v1.5.0
Run dbeaec1c was finished

Despite this, in testomat.io, the run remains in progress :

[olexandr13]

@florianlefeuvre could you reporoduce this problem locally?

[TetianaKhomenko]

Hi @MathieuJouis @florianlefeuvre! Does the issue still reproduce?

[MathieuJouis]

Hi @TetianaKhomenko ,

Unfortunately yes.

I tried with version 1.5.1 of the reporter, but the problem persists

[TetianaKhomenko]

@AZANIR could you please check this on the latest reporter version?

[MathieuJouis]

Hello @olexandr13 ,

We recently tried to use the latest version (1.6.5) of @testomatio/reporter but we are facing a new issue in CI job.

Please see error below :

An error was thrown in your plugins file while executing the handler for the after:spec event.
The error we received was:
TypeError: Cannot read properties of null (reading 'path')
    at Client.addTestRun (/node_modules/@testomatio/reporter/lib/client.js:184:16)
    at Object.handler (/node_modules/@testomatio/reporter/lib/adapter/cypress-plugin/index.js:87:16)
    at invoke (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:185:18)
    at /root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/util.js:59:14
    at tryCatcher (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/bluebird/js/release/method.js:39:29)
    at Object.wrapChildPromise (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/util.js:58:23)
    at RunPlugins.execute (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:164:21)
    at EventEmitter.<anonymous> (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_plugins.js:56:12)
    at EventEmitter.emit (node:events:524:28)
    at EventEmitter.emit (node:domain:489:12)
    at process.<anonymous> (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/util.js:33:22)
    at process.emit (node:events:524:28)
    at process.emit (node:domain:489:12)
    at process.emit.sharedData.processEmitHook.installedValue [as emit] (/root/.cache/Cypress/13.16.1/Cypress/resources/app/node_modules/@cspotcode/source-map-support/source-map-support.js:745:40)
    at emit (node:internal/child_process:949:14)
    at processTicksAndRejections (node:internal/process/task_queues:91:21)

This error occurs after the completion of a single feature file (even with all tests succeeded).

[olexandr13]

@MathieuJouis thanks for report, will fix today

[florianlefeuvre]

Hi @olexandr13, thanks for the fast replies. We're currently working on this subject along with @MathieuJouis.

For the moment, we did fallback on @testomatio/[email protected] and used the first strategy on your documentation, it seems to be working fine without TESTOMATIO_PROCEED variable as @DavertMik mentioned in his post.

  variables:
    TESTOMATIO: ${TESTOMATIO_TOKEN}
    TESTOMATIO_TITLE: 'Report ${env} at ${CI_PIPELINE_CREATED_AT} (pipeline: #${CI_PIPELINE_ID})'
    TESTOMATIO_ENV: ${env}
  script:
    - sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/' /etc/ssl/openssl.cnf
    - npx start-test-run -c "npm run parallel:$env$SEVERITY_LEVEL -- --threads ${THREADS_NUMBER}"

---

As for the error of @testomatio/[email protected] explained by @MathieuJouis, it seems to comes from manuallyAttachedArtifacts array from client.jsfile, line 178 that contains as many "undefined" values as there are success tests.

https://github.com/testomatio/reporter/blob/87e229d8d796d18352d005dcb397424cc88bdc41/lib/client.js#L178-L190

The iteration line 190 seems to push those undefined further, triggering errors I guess. Also, the condition typeof f === 'object' would resolve as true is f is null, as typeof null is 'object' (maybe that's a case that never occurs, though).

We will wait for your correction before updating to 1.6.x then !

[olexandr13]

@florianlefeuvre sorry for delay. fixed in 1.6.6