wdio-jasmine-framework
wdio-jasmine-framework copied to clipboard
Does not retry when `expect()` fails
Hi there,
I am building a test suite using this adapter, and can't get my jasmine tests to retry when expectations fail. Throwing an error explicitly works though (as explained in #16 by @christian-bromann ):
package.json
{
"dependencies": {
"jasmine": "2.5.3",
"selenium-standalone": "6.1.0",
"wdio-jasmine-framework": "0.3.0",
"wdio-selenium-standalone-service": "0.0.8",
"webdriverio": "4.6.2"
}
}
wdio.config.js
exports.config = {
specs: ['./tests.js'],
services: ['selenium-standalone'],
capabilities: [{
browserName: 'chrome',
}],
framework: 'jasmine',
};
tests.js
describe('failing tests', () => {
it('b', () => {
console.log('run me b');
throw new Error('fail');
}, 2);
it('a', () => {
console.log('run me a');
expect(true).toBe(false);
}, 2);
});
console output
run me b
run me b
run me b
run me a
0 passing (3.00s)
2 failing
I tried throwing errors in the expectationResultHandler
instead, but that doesn't help either. Any help would be greatly appreciated! 🙂
Looks like retry only works if there is an exception (https://github.com/webdriverio/wdio-sync/blob/32bf88aa30373e50529a2b9cae58573c0a5e6a07/index.js#L472) so presumably expect
is not throwing? Should it throw?
Looking at how Jasmine works, it looks like expect
is not designed to throw—it just mutates a shared object with the results
(failedExpectations
): https://jasmine.github.io/2.3/custom_reporter.html
Can we update this adapter so it retries if result.failedExpectations.length > 0
on the object returned from it
?
Can we update this adapter so it retries if result.failedExpectations.length > 0 on the object returned from it?
If this solves the issue above..sure!
I'm not quite sure how to do that, though. 😒 If I understand correctly, this adapter invokes executeSync
which is responsible for doing the retry, but I'm not sure where/how to access the Jasmine Spec
object returned by it
which contains the details about failed expectationss.
access the Jasmine Spec object returned by it which contains the details about failed expectationss.
it is in the wdio-sync
project. Please make sure that the changes apply only for jasmine or are compatible with Mocha and Cucumber
Is this the place to check? https://github.com/webdriverio/wdio-sync/blob/32bf88aa30373e50529a2b9cae58573c0a5e6a07/index.js#L592
Look at the object returned from origFn
? If some heuristic is true, recurse?
yes
this seems to be more tricky than expected. The expect indeed doesn't throw an error. There is no way to detect if the spec fails or not. I actually don't really know how to work around that.