[🐛 Bug]: No reports if session has already terminated when test fails

[stevoland]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

WebdriverIO Version

8.18.2

Node.js Version

20.13.1

Mode

Standalone Mode

Which capabilities are you using?

No response

What happened?

We sometimes receive "Session not started or terminated" responses from Browserstack.

In these cases no report is written.

This is due to the uncaught exception "invalid session id: A session is either terminated or not started" at https://github.com/webdriverio/webdriverio/blob/16434613fd3dc04667e789bdf276f595552a62e3/packages/wdio-runner/src/index.ts#L451, preventing shutdown from being called

Catching the exception and continuing seems to fix the reports

try {
    // This will throw if a session is already terminated preventing
    // reporters from receiving "runner:end" event
    await this._browser?.deleteSession();
} catch (error) {
    log.error('Error deleting session', error);
}

What is your expected behavior?

No response

How to reproduce the bug.

Sorry, no repro.

I found I was able to reproduce by waiting over 5 minutes before throwing in a test but this is likely something specific to our setup

import { Then } from '@wdio/cucumber-framework'

Then('I blow up very slowly', { timeout: 6 * 60 * 1000 }, async function () {
  await new Promise((resolve) => setTimeout(resolve, 5.1 * 60 * 1000))
  throw new Error('boom')
}

Relevant log output

2024-07-02T12:33:45.283Z WARN webdriver: Request failed with status 200 due to Unable to communicate to node
2024-07-02T12:33:45.859Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:33:45.864Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:33:45.869Z ERROR webdriver: Request failed with status 404 due to WebDriver Error: Session not started or terminated
2024-07-02T12:35:10.937Z WARN webdriver: Request failed with status 200 due to Session not started or terminated
2024-07-02T12:35:11.511Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:35:12.130Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:35:12.712Z ERROR webdriver: Request failed with status 404 due to WebDriver Error: Session not started or terminated
2024-07-02T12:35:43.356Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:35:44.514Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:35:45.101Z WARN webdriver: Request failed with status 404 due to Session not started or terminated
2024-07-02T12:35:45.694Z ERROR webdriver: Request failed with status 404 due to WebDriver Error: Session not started or terminated
2024-07-02T12:35:45.695Z ERROR @wdio/utils:shim: WebDriver Error: Session not started or terminated
    at getErrorFromResponseBody (node_modules/webdriver/build/utils.js:194:12)
    at NodeJSRequest._request (node_modules/webdriver/build/request/index.js:164:23)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Browser.wrapCommandFn (node_modules/@wdio/utils/build/shim.js:81:29)
    at async Browser.wrapCommandFn (node_modules/@wdio/utils/build/shim.js:81:29)
    at async BrowserstackService._executeCommand (node_modules/@wdio/browserstack-service/build/service.js:353:17)
    at async BrowserstackService.beforeScenario (node_modules/@wdio/browserstack-service/build/service.js:206:9)
    at async Promise.all (index 4)
    at async executeHooksWithArgsShim (node_modules/@wdio/utils/build/shim.js:59:20)
    at async createCustomWorld.<anonymous> (node_modules/@wdio/cucumber-framework/build/index.js:289:13)

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• [X] I have searched the existing issues

[BorisOsipov]

In these cases no report is written.

There are dozens wdio reporters. What kind of reporter you tell about?

[stevoland]

There are dozens wdio reporters. What kind of reporter you tell about?

All of them I imagine. If you read the issue, "runner:end" event from @wdio/reporter is not emitted so no reporter can report.

[edit]

For instance: @wdio/junit-reporter, @wdio/spec-reporter, anything that extends @wdio/reporter

[BorisOsipov]

Thank you. Examples are very important. The runner:end event is not used by all reporters.

[wdio-bot]

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

[christian-bromann]

I think all reporter should report that the test didn't even started. Any contributions are appreciated!

[stevoland]

I think all reporter should report that the test didn't even started

The test has started but, spec and junit reporters (at least) don't write until "runner:end" so in the case where a session has already terminated for whatever reason no report is written.

Any contributions are appreciated

I'm happy to raise a PR with the fix proposed here if there's no argument against it?

[christian-bromann]

I'm happy to raise a PR with the fix proposed here if there's no argument against it?

Awesome, please go ahead!