Different quarantine attempts sometimes have equal testRunId's

[VasilyStrelyaev]

What is your Scenario?

I run a specific test suite with quarantine mode enabled and a custom reporter attached.

What is the Current behavior?

In the reporter methods reportTestActionDone (the actionInfo.testRunId argument) and reportTestDone (the testRunInfo.quarantine[testRunId] dictionary), TestCafe passes the same testRunId for all the quarantine attempts.

What is the Expected behavior?

Each quarantine attempt has its own testRunId, just like it normally happens.

What is your public website URL? (or attach your complete example)

See steps to reproduce

What is your TestCafe test code?

See steps to reproduce

Your complete configuration file

No response

Your complete test report

No response

Screenshots

No response

Steps to Reproduce

Reproduced on DevExtreme TestCafe tests.

Prerequisites

Install

• .NET Core LTS (3.1 currently)
• .NET Core Latest (6.x I think)

Prepare a debugging reporter

It's good to console.log either actionInfo.testRunId on reportTestActionDone, testRunInfo.quarantine on reportTestDone, or testRunInfo.browsers on reportTestDone:

module.exports = function () {
    return {
        noColors: true,
        
        reportTaskStart (/* startTime, userAgents, testCount */) {
        },

        reportFixtureStart (/* name, path */) {
        },

        reportTestStart () {
        },

        reportTestActionDone () {
        },

        reportTestDone (name, testRunInfo) {
            const { browsers } = testRunInfo;

            console.log(`Browsers: ${JSON.stringify(browsers)}`);
        },

        reportTaskDone (/* endTime, passed, warnings */) {
        }
    };
};

Create a project testcafe-reporter-stab (with package.json) with such code locally, and link it to the local package storage with npm link.

Steps

I can only reproduce this behavior when running a big part of the test suite. So, here are the steps:

1. Clone my fork of the DevExtreme repo: https://github.com/VasilyStrelyaev/DevExtreme

2. Check out the integrate-dashboard branch

3. npm install

4. npm link testcafe-reporter-stab

5. npm run build:dev

6. npx gulp localization

7. Open testing/testcafe/runner.js locally and specify your reporter's name on line 116.

   reporter: 'stab',
   

8. Now run the tests: npm run test-testcafe -- --concurrency 2 --browsers=chrome:headless --componentFolder dataGrid --quarantineMode true

Examine the output

If you use the suggested reporter that outputs browsers, you will see a lot of records like:

Browsers: [{"testRunId":"Qpi6KfgXf","name":"Chrome","version":"103.0.5060.134","platform":"desktop","os":{"name":"macOS","version":"10.15.7"},"engine":{"name":"Blink","version":"0.0"},"prettyUserAgent":"Chrome 103.0.5060.134 / macOS 10.15.7","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/103.0.5060.134 Safari/537.36","alias":"chrome:headless","headless":true,"quarantineAttemptsTestRunIds":["THqmFHYCe","5I2k6sOVc","Qpi6KfgXf"]}]

There will be good records, like the one above, with distinct quarantineAttemptsTestRunIds:

"quarantineAttemptsTestRunIds":["THqmFHYCe","5I2k6sOVc","Qpi6KfgXf"]

And bad ones, like:

"quarantineAttemptsTestRunIds":["aom8W2Gj8","aom8W2Gj8","aom8W2Gj8"]

TestCafe version

1.20.0

Node.js version

No response

Command-line arguments

See steps

Browser name(s) and version(s)

No response

Platform(s) and version(s)

No response

Other

No response

[VasilyStrelyaev]

UPD:

Sometimes (most of the time), I have the same testRunId for different testId. This does not cause a problematic behavior in the Dashboard, because testId are different, but is it intended?

(marker mine, the screenshot taken with concurrency=1)

[Artem-Babich]

We researched this issue and found that the same testRunId values are returned for tests that use the disablePageReloads property. disablePageReloads is an experimental and undocumented feature that prevents page reloads and session re-creation between tests. The testRunId property is actually the session ID. Therefore, different test runs have the same testRunId.