Parallel runs on Browserstack are interdependent and not concurrent

[chheller]

When running tests with concurrency on Browserstack, the tests only actually run sequentially. This screenshot demonstrates the parallel tests (most recent) run for approximately the same amount of time as the tests with no concurrency (least recent).

When viewing the videos of the tests, it is clear that the parallel sessions wait for each other to finish running a single test a time. This leads to the browser not being opened for >30s on one session, while the other opens almost immediately. After that test finishes, this session hangs on Testcafe's loading screen while the other session finishes its test.

testcafe 0.21.0 node: 8.9.1 testcafe-browser-provider-browserstack 1.5.0 Browsers: Firefox 61.0 Chrome 68.0 Edge 17.0 Run on Mac OS X High Sierra 10.13.6

[AndreyBelym]

Hi @chheller! I don't think that the problem is in the provider. Try to run the following test with and without concurrency:

fixture `test`.page`example.com`;

const DELAY = 30000;
const COUNT = 6;

for(let i = 0; i < COUNT; i++)
    test(`${i}`, t => t.wait(DELAY))

On my machine, test takes 3m05s to run without concurrency, and 1m35s with the -c 2 option.

I think there is something in your test code that blocks parallel test execution, could you plese share it with me?

[chheller]

Hi @AndreyBelym

It could be due to the way I've structured our tests, though I notice this isn't an issue when running tests locally.

import environment from '../config/environment';
import { TestSuite } from '../support/commands';
import { loginByApi } from '../support/workflow/login-page';
import loginSpecs from '../support/specs/login';
import shipmentSpecs from '../support/specs/book-shipment';

const { baseUrl } = environment;
fixture`Smoke Suite`;
fixture`Login Suite`.page(baseUrl);

const loginTestSuite = new TestSuite(loginSpecs);
loginTestSuite.run();

fixture`Shipment Suite`.page(`${baseUrl}/#/quotes`).beforeEach(async () => {
  await loginByApi();
});

const shipmentTestSuite = new TestSuite(shipmentSpecs);
shipmentTestSuite.run();
// test() hack around https://github.com/DevExpress/testcafe/issues/2074

The TestSuite class loads tests exported from definition files that looks like

export default {
  'Should successfully login': async (t: TestController) => {
    await t.testcodehere()
  }
};

Given an object of shape

{[key: string]: (t: TestController) => Promise<void>

the TestSuite class will map its keys to an array of function wrappers which wrap testcafe's test function so that tests can be dynamically loaded in a test suite file. I'd suspect that's the most likely to introduce a bug, however I don't see the same pseudo-parallel behavior when I run the tests against local browsers concurrently.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[github-actions[bot]]

We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.