throw error in case of failure for programatic api

[swrdfish]

Attempt to fix #2510

[tim-yao]

Hi @swrdfish, any idea on this? I hope this PR won't be stalled. Really need a fix. Thanks!

[swrdfish]

@tim-yao the fix is not as simple as I thought it would be. Hopefully we'll have some update by early next week.

[beatfactor]

The only way to do this in a safe manner is to introduce a new settings which will be turned off by default. Something like the always_async_commands settings.

[tim-yao]

Hi all. Happy to help on this if I can get some guidance. Really need this, as the issue makes the programmatic API basically useless.

[beatfactor]

Hi all. Happy to help on this if I can get some guidance. Really need this, as the issue makes the programmatic API basically useless.

I think it should work fine if you set always_async_commands to true, which is the case in the cucumber runner.

[tim-yao]

Hi @beatfactor I added always_async_commands but still the same. See this commit: https://github.com/nightwatchjs-community/cucumber-nightwatch/pull/9/commits/103ae61a2fa1ca1dd583fc40f3402cb74c715814 And test result: https://github.com/nightwatchjs-community/cucumber-nightwatch/actions/runs/3745075953/jobs/6359112950 Any idea?

[tim-yao]

Hi all. Happy to help on this if I can get some guidance. Really need this, as the issue makes the programmatic API basically useless.

I think it should work fine if you set always_async_commands to true, which is the case in the cucumber runner.

Did some debugging today and I think the cucumber runner works because of this additional logic: https://github.com/nightwatchjs/nightwatch/blob/main/lib/core/asynctree.js#L95-L97

Should we add something like:

    if (this.programaticAPI) {
      return err.waitFor;
    }

[tim-yao]

There are two more related issues I found in the programmatic API. They can be reproduced at https://github.com/nightwatchjs/cucumberjs-boilerplate.

1. abortOnFailure option is not working. The below examples will fail the test step even though abortOnFailure is false.

await browser.waitForElementPresent('.not-existing-element', 100, false)
await browser.waitForElementPresent({
      selector: '.not-existing-element',
      retryInterval: 1000,
      abortOnFailure: false
    });

2. click command won't fail. await browser.click('#not-existing-element') won't fail the step.