nightwatch skip_testcase_on_fail Does not skip the remaning test case on failure.

Description of the bug/issue

Context

There are test-suite use cases that can achieved through a combination of abortOnFailure and skip_test_on_fail. abortOnFailure is to determine whether to end the execution of the test case at that node or continue the execution of the test case. The flag does not affect marking a test as a pass or execution of subsequent test cases. Whereas skip_testcase_on_failure will allow us to decide to skip the following test cases in the test suite if a failure occurs in the previous test case.

The bug is created when abortOnFailure is false and skip_testcase_on_fail is set to true. Upon failure, the rest of the test cases are not skipped. This can be reproduced with the example shown below.

Steps to reproduce

Run the test suite shown below with nightwatch.

Sample test

module.exports = {
  'wait for not': () => {
    browser.url('https://www.nightwatchjs.org');
    browser.waitForElementNotVisible('#header', 1000, 100, false) //failure
    browser.pause(1);
  },
  'wait for': () => {
    browser.url('https://www.nightwatchjs.org');
    browser.setAttribute('nav.navigation', 'hidden', 'true');
    browser.waitForElementVisible('nav.navigation', 1000, 100, false); 
  }
};

Command to run

npx nightwatch example.js --env chrome --verbose

Verbose Output

Starting ChromeDriver on port 9515...
   Request POST /session  
   {
     capabilities: {
       firstMatch: [ {} ],
       alwaysMatch: {
         browserName: 'chrome',
         webSocketUrl: true,
         'goog:chromeOptions': {
           w3c: true,
           args: [ '--remote-debugging-port=0' ],
           binary: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
         }
       }
     }
⠋ Starting ChromeDriver on port 9515...
   Response 200 POST /session (1546ms)
   {
     value: {
       capabilities: {
         acceptInsecureCerts: false,
         browserName: 'chrome',
         browserVersion: '118.0.5993.70',
         chrome: {
           chromedriverVersion: '118.0.5993.70 (e52f33f30b91b4ddfad649acddc39ab570473b86-refs/branch-heads/5993@{#1216})',
           userDataDir: '/var/folders/n9/8h4bw85j2y95c2rw67sjjdmr0000gn/T/.org.chromium.Chromium.VXtmxY'
         },
         'fedcm:accounts': true,
         'goog:chromeOptions': { debuggerAddress: 'localhost:58181' },
         networkConnectionEnabled: false,
         pageLoadStrategy: 'normal',
         platformName: 'mac',
         proxy: {},
         setWindowRect: true,
         strictFileInteractability: false,
         timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
         unhandledPromptBehavior: 'dismiss and notify',
         webSocketUrl: 'ws://localhost:9515/session/26feabcc566d0e35e8545d6ead0a09ef',
         'webauthn:extension:credBlob': true,
         'webauthn:extension:largeBlob': true,
         'webauthn:extension:minPinLength': true,
         'webauthn:extension:prf': true,
         'webauthn:virtualAuthenticators': true
       },
       sessionId: '26feabcc566d0e35e8545d6ead0a09ef'
     }
ℹ Connected to ChromeDriver on port 9515 (1714ms).
  Using: chrome (118.0.5993.70) on MAC.

 Received session with ID: 26feabcc566d0e35e8545d6ead0a09ef

 → Running [before]:
 → Completed [before].



  Running wait for not:
───────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].
 
 → Running command: url ('https://www.nightwatchjs.org')
  ⠋ Loading url: https://www.nightwatchjs.org
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/url  
  ⠙ Loading url: https://www.nightwatchjs.org
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/url (2559ms)
  ℹ Loaded url https://www.nightwatchjs.org in 2563ms
 → Completed command: url ('https://www.nightwatchjs.org') (2570ms)
 
 → Running command: waitForElementNotVisible ('#header', 1000, 100, false)
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements  
   { using: 'css selector', value: '#header' }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements (17ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (17ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (13ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (19ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (20ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (12ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
  ✖ NightwatchAssertError
   Timed out while waiting for element <#header> to not be visible for 1000 milliseconds. - expected "not visible" but got: "visible" (1003ms)

    Error location:
    /Users/ravi/Documents/Projects/nightwatch/examples/tests/ecosia.js:
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     2 |   'wait for not': () => {
     3 |     browser.url('https://www.nightwatchjs.org');
     4 |     browser.waitForElementNotVisible('#header', 1000, 100, false) 
     5 |     browser.pause(1);
     6 |   },
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

 → Completed command: waitForElementNotVisible ('#header', 1000, 100, false) (1011ms)
 
 → Running command: pause (1)
 → Completed command: pause (1) (2ms)
 → Running [afterEach]:
 → Completed [afterEach].

  FAILED: 1 assertions failed (3.594s)



  Running wait for:
───────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].
 
 → Running command: url ('https://www.nightwatchjs.org')
  ⠋ Loading url: https://www.nightwatchjs.org
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/url  
  ⠙ Loading url: https://www.nightwatchjs.org
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/url (142ms)
  ℹ Loaded url https://www.nightwatchjs.org in 145ms
 → Completed command: url ('https://www.nightwatchjs.org') (145ms)
 
 → Running command: setAttribute ('nav.navigation', 'hidden', 'true')
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements  
   { using: 'css selector', value: 'nav.navigation' }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements (11ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36'
       }
     ]
  }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: 'var passedArgs = Array.prototype.slice.call(arguments,0); return (function (e, a, v) { try {\n' +
       "                        if (e && typeof e.setAttribute == 'function') {\n" +
       '                            e.setAt... (480 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36'
       },
       'hidden',
       'true'
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (8ms)
   { value: true }
 → Completed command: setAttribute ('nav.navigation', 'hidden', 'true') (24ms)
 
 → Running command: waitForElementVisible ('nav.navigation', 1000, 100, false)
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements  
   { using: 'css selector', value: 'nav.navigation' }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements (7ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36'
       }
     ]
  }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36',
         ELEMENT: '66491573582C4293AF6020DBE7C7A88E_element_36'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (10ms)
   { value: true }
  ✔ Element <nav.navigation> was visible after 19 milliseconds.
 → Completed command: waitForElementVisible ('nav.navigation', 1000, 100, false) (20ms)
 → Running [afterEach]:
 → Completed [afterEach].

  ✨ PASSED. 1 assertions. (192ms)
 → Running [after]:
 → Completed [after].
 
 → Running command: end (true)
 
 → Running command: session ('delete', [Function])
   Request DELETE /session/26feabcc566d0e35e8545d6ead0a09ef  

   Response 200 DELETE /session/26feabcc566d0e35e8545d6ead0a09ef (77ms)
   { value: null }
 → Completed command: session ('delete', [Function]) (82ms)
 Wrote log file to: /Users/ravi/Documents/Projects/nightwatch/logs/ecosia_chromedriver.log
 → Completed command: end (true) (93ms)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (6.08s):  
   - 1 assertions failed; 1 passed

   ✖ 1) ecosia

   – wait for not (3.594s)

   → ✖ NightwatchAssertError
   Timed out while waiting for element <#header> to not be visible for 1000 milliseconds. - expected "not visible" but got: "visible" (1003ms)

    Error location:
    /Users/ravi/Documents/Projects/nightwatch/examples/tests/ecosia.js:
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     2 |   'wait for not': () => {
     3 |     browser.url('https://www.nightwatchjs.org');
     4 |     browser.waitForElementNotVisible('#header', 1000, 100, false) 
     5 |     browser.pause(1);
     6 |   },
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––


 ChromeDriver process closed.
 Wrote HTML report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/nightwatch-html-report/index.html

 Wrote JSON report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/CHROME_118.0.5993.70__ecosia.json
 Wrote Rerun Json report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/minimal_report.json
 Wrote XML report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/CHROME_118.0.5993.70__ecosia.xml

Nightwatch Configuration

No response

Nightwatch.js Version

3.2.1

Node Version

16

Browser

No response

Operating System

No response

Additional Information

No response

Oct 19 '23 08:10 gravityvi

@gravityvi This issue is still open or not..

Feb 23 '24 14:02 Kartikeya-09

I'd like to work on this issue. @gravityvi

Aug 02 '24 18:08 BishalJena

Hey @BishalJena 👋🏻 , sure go ahead. I have not contributed since a while now so I might have lost context but @garg3133 can definitely help here.

Aug 04 '24 09:08 gravityvi