tryTo() triggering false positive to pauseOnFail - stopping at first scenario

[incoming-th]

What are you trying to achieve?

Trying to run 2 scenarios with one containing tryTo().

What do you get instead?

With the flag -p pauseOnFail, codeceptjs is stopping after scenario 1, even when no error or failure happens.

I believe this is a false positive triggered by tryTo() as seeing in the verbose:

[1] <tryTo> Error (Non-Terminated) | Error: Element "NOPE" is not visible on page. | (err) => { step.status = 'failed'; step.endTime =

Provide console output if related. Use --verbose mode for more details.

> npx codeceptjs run ./tests/test.js --verbose -p pauseOnFail

***************************************
nodeInfo:  18.20.4
osInfo:  Linux 5.15 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
cpuInfo:  (32) x64 Intel(R) Core(TM) i9-14900HX
chromeInfo:  Not Found
edgeInfo:  "N/A"
firefoxInfo:  undefined
safariInfo:  N/A
If you need more detailed info, just run this: npx codeceptjs info
***************************************
CodeceptJS v3.6.6 #StandWithUkraine
Using test root "xxx"
Helpers: Playwright, Hooks
Plugins: screenshotOnFail, tryTo, retryFailedStep, retryTo, eachElement, pauseOnFail

Fixed Assets --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  01 - Test
    I am on page "https://github.com/login"
      I am on page "https://github.com/login"
    Info: RetryFailedStep plugin is disabled inside tryTo block
    I see element "NOPE"
      I see element "NOPE"
    [1] <tryTo>  Error (Non-Terminated) | Error: Element "NOPE" is not visible on page. | (err) => { step.status = 'failed'; step.endTime = ...
 › Unsuccessful try > Error: Element "NOPE" is not visible on page.
  ✔ OK in 3961ms

 Interactive shell started
 Use JavaScript syntax to try steps in action
 - Press ENTER to run the next step
 - Press TAB twice to see all available commands
 - Type exit + Enter to exit the interactive shell
 - Prefix => to run js commands 
 I.

Provide test source code if related

Feature('Fixed Assets');

Scenario('01 - Test',  async ({ I }) => {
  await I.amOnPage('https://github.com/login');
  await tryTo(() => I.seeElement('NOPE'))
});

Scenario('02 - Test',  async ({ I }) => {
  await I.amOnPage('https://github.com/login');
});

Details

• CodeceptJS version: 3.6.6
• NodeJS Version: 18.20.4
• Operating System: Ubuntu 22.04.4
• Playwright
• Configuration file:

require('dotenv').config();
const http = require('http');
require('./heal');
const { setHeadlessWhen, setCommonPlugins } = require('@codeceptjs/configure');

// Turn on headless mode when running with HEADLESS=true environment variable
// Export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// Enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

/** @type {CodeceptJS.MainConfig} */
exports.config = {
  name: 'xxx',
  // For local development we rename the files we want to test
  tests: './tests/*.js',
  output: './output',
  helpers: {
    Playwright: {
      browser: 'chromium',
      url: 'http://localhost',
      // Default time for all wait* actions
      waitForTimeout: 60000,
      // Time to wait after each action
      waitForAction: 1000,
      // Highlight element only work with --verbose
      highlightElement: true,
      show: true,
      video: true,
      keepVideoForPassedTests: true,
      windowSize: '1024x768',
      recordVideo: { // Seems to not work but required to use windowSize -- solved in 3.6.2
        size: {
          width: 1024,
          height: 768
        }
      },
      trace: false
    },
    Hooks: {
      require: './events/hooks.js'
    },
  },
}

[incoming-th]

Tried to look at the code but I cannot find where this is triggered.

[incoming-th]

Still looking for a solution on this issue but I cannot find where is the trigger.

[mirao]

Got the same issue with CodeceptJS 3.6.10 An unsuccessful try was handled correctly by a test (the step and scenario didn't fail), but I got an interactive mode at the end of the scenario because of pauseOnFail 🐞 .

codeceptjs run InstrumentOnlineQuizMultipleChoice_test.ts -p pauseOnFail --profile dev --verbose

  Create a lesson
 › Test Timeout: 600s
 › [Dir] /home/mirao/workspace/codeceptjs/tests/s4w
    I click {xpath: //button[contains(., 'Add Lesson')]}
    I fill field .form input, "ATLU511218139137517"
    I fill field {xpath: //input[@name = 'name']}, "ATL203822590223859"
    I click {xpath: //textarea[@name = 'lesson_objectives']}
    I type "AT Lesson Objective"
    I click {xpath: //button[contains(., 'Create')]}
    I see element {xpath: //*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-treenode ')][starts-with(@class, "key")]//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-node-content-wrapper ')][contains(., 'ATLU511218139137517')]}
    I grab number of visible elements {xpath: //*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-treenode ')][starts-with(@class, "key")]//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-node-content-wrapper ')][contains(., 'ATL203822590223859')]}
    Info: RetryFailedStep plugin is disabled inside tryTo block
    I click {xpath: (//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-switcher_close ')])[position()=last()-0]}
    [1] <tryTo>  Error (Non-Terminated) | Error: Clickable element "{"type":"xpath","output":null,"strict":true,"locator":{"xpath":"(//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-switcher_close ')])[position()=last()-0]"},"value":"(//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-switcher_close ')])[position()=last()-0]"}" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
 › Unsuccessful try > Error: Clickable element "{"type":"xpath","output":null,"strict":true,"locator":{"xpath":"(//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-switcher_close ')])[position()=last()-0]"},"value":"(//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-switcher_close ')])[position()=last()-0]"}" was not found by text|CSS|XPath
    I click {xpath: //*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-treenode ')][starts-with(@class, "key")]//*[@class and contains(concat(' ', normalize-space(@class), ' '), ' rc-tree-node-content-wrapper ')][contains(., 'ATL203822590223859')]}
    I see text equals "ATL203822590223859", .ipoLUName
  ✔ OK in 3798ms

 Interactive shell started
 Use JavaScript syntax to try steps in action
 - Press ENTER to run the next step
 - Press TAB twice to see all available commands
 - Type exit + Enter to exit the interactive shell
 - Prefix => to run js commands 
 I.

If I remove tryTo() from my code, then everything works as expected.

[incoming-th]

CodeceptJS 3.7.5 and this issue still persists.

tryTo() + -p pauseOnFail will stop the Scenario() even if there are other to run.

Could be related to plugin pauseOnFail reading some "Error" substring from tryTo:

[1] <tryTo> Error (Non-Terminated) | Error: Element "NOPE" is not visible on page. | err => { step.status = 'failed' step.endTime = +Da...

@kobenguyent Sorry to tag you but did you ever face this issue during tests?