Does not retry when `expect()` fails

[samijaber]

Hi there,

I am building a test suite using this adapter, and can't get my jasmine tests to retry when expectations fail. Throwing an error explicitly works though (as explained in #16 by @christian-bromann ):

package.json

{
  "dependencies": {
    "jasmine": "2.5.3",
    "selenium-standalone": "6.1.0",
    "wdio-jasmine-framework": "0.3.0",
    "wdio-selenium-standalone-service": "0.0.8",
    "webdriverio": "4.6.2"
  }
}

wdio.config.js

exports.config = {
  specs: ['./tests.js'],
  services: ['selenium-standalone'],
  capabilities: [{
    browserName: 'chrome',
  }],
  framework: 'jasmine',
};

tests.js

describe('failing tests', () => {
  it('b', () => {
    console.log('run me b');
    throw new Error('fail');
  }, 2);

  it('a', () => {
    console.log('run me a');
    expect(true).toBe(false);
  }, 2);
});

console output

run me b
run me b
run me b
run me a

0 passing (3.00s)
2 failing

I tried throwing errors in the expectationResultHandler instead, but that doesn't help either. Any help would be greatly appreciated! 🙂

[OliverJAsh]

Looks like retry only works if there is an exception (https://github.com/webdriverio/wdio-sync/blob/32bf88aa30373e50529a2b9cae58573c0a5e6a07/index.js#L472) so presumably expect is not throwing? Should it throw?

Looking at how Jasmine works, it looks like expect is not designed to throw—it just mutates a shared object with the results (failedExpectations): https://jasmine.github.io/2.3/custom_reporter.html

Can we update this adapter so it retries if result.failedExpectations.length > 0 on the object returned from it?

[christian-bromann]

Can we update this adapter so it retries if result.failedExpectations.length > 0 on the object returned from it?

If this solves the issue above..sure!

[OliverJAsh]

I'm not quite sure how to do that, though. 😒 If I understand correctly, this adapter invokes executeSync which is responsible for doing the retry, but I'm not sure where/how to access the Jasmine Spec object returned by it which contains the details about failed expectationss.

[christian-bromann]

access the Jasmine Spec object returned by it which contains the details about failed expectationss.

it is in the wdio-sync project. Please make sure that the changes apply only for jasmine or are compatible with Mocha and Cucumber

[OliverJAsh]

Is this the place to check? https://github.com/webdriverio/wdio-sync/blob/32bf88aa30373e50529a2b9cae58573c0a5e6a07/index.js#L592

Look at the object returned from origFn? If some heuristic is true, recurse?

[christian-bromann]

yes

[christian-bromann]

this seems to be more tricky than expected. The expect indeed doesn't throw an error. There is no way to detect if the spec fails or not. I actually don't really know how to work around that.