a failing async test causes any browser object calls to fail in the afterEach step

[chris-jackson-actionqa]

Describe the bug

When using async for the test, the afterEach hook can hang if that tests fails. It only hangs if you invoke the browser object. In my example below, I'm just pausing for one second.

If you remove the async from the test, the afterEach hook runs fine. If you remove the browser.pause call from the afterEach, it also runs fine. If the test passes, then it runs fine.

When running the test, you'll see that the console.log('Made it here') never gets invoked. Then the afterEach call will time out.

Sample test

sampleTest.js

// Please add the sample test here

describe('repro afterEach', () => {
  afterEach(async (browser, done) => {
    await browser.pause(1_000);
    console.log('made it here');
    browser.end(done);
  });

  it('fail on purpose', async (browser) => {
    browser.url('http://example.com/');
    browser.expect.element('body > div > h1').text.to.contain('blah');
  });
});

Run with command

$ nightwatch test/sampleTest.js -e chrome

Verbose output

debug.log

<!-- Include the verbose output, if possible (run nightwatch with `--verbose` argument) -->

❯ npx nightwatch -e chrome --verbose

[repro afterEach] Test Suite
────────────────────────────────────────────────────────
⠋ Starting ChromeDriver on port 9515...
 Starting ChromeDriver with server_path=/Users/cjackson/Documents/nwrepro/node_modules/chromedriver/lib/chromedriver/chromedriver...
   Request POST /session  
   {
     desiredCapabilities: {
       browserName: 'chrome',
       'goog:chromeOptions': { w3c: true, args: [] },
       name: 'repro afterEach'
     },
     capabilities: {
       alwaysMatch: {
         browserName: 'chrome',
         'goog:chromeOptions': { w3c: true, args: [] }
       }
     }
⠦ Starting ChromeDriver on port 9515...
   Response 200 POST /session (1316ms)
   {
     value: {
       capabilities: {
         acceptInsecureCerts: false,
         browserName: 'chrome',
         browserVersion: '101.0.4951.64',
         chrome: {
           chromedriverVersion: '101.0.4951.41 (93c720db8323b3ec10d056025ab95c23a31997c9-refs/branch-heads/4951@{#904})',
           userDataDir: '/var/folders/91/l15sfgjs7cg2d0htqwh6h93w0000gn/T/.com.google.Chrome.uQRJ6L'
         },
         'goog:chromeOptions': { debuggerAddress: 'localhost:52475' },
         networkConnectionEnabled: false,
         pageLoadStrategy: 'normal',
         platformName: 'mac os x',
         proxy: {},
         setWindowRect: true,
         strictFileInteractability: false,
         timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
         unhandledPromptBehavior: 'dismiss and notify',
         'webauthn:extension:credBlob': true,
         'webauthn:extension:largeBlob': true,
         'webauthn:virtualAuthenticators': true
       },
       sessionId: '370fe688bdb4bc37e3c0514f4418a4a3'
     }
ℹ Connected to ChromeDriver on port 9515 (1383ms).
  Using: chrome (101.0.4951.64) on MAC OS X.

 Received session with ID: 370fe688bdb4bc37e3c0514f4418a4a3

 → Running [before]:
 → Completed [before].



  Running fail on purpose:
───────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].
 → Running [afterEach]:
 
 → Running command: url ('http://example.com/')
  ⠋ Loading url: http://example.com/
   Request POST /session/370fe688bdb4bc37e3c0514f4418a4a3/url  
  ⠴ Loading url: http://example.com/
   Response 200 POST /session/370fe688bdb4bc37e3c0514f4418a4a3/url (415ms)
  ℹ Loaded url http://example.com/ in 416ms
  → Completed command: url ('http://example.com/') (417ms)
 
 → Running command: expect.element ('body > div > h1')
   Request POST /session/370fe688bdb4bc37e3c0514f4418a4a3/elements  
   { using: 'css selector', value: 'body > div > h1' }
   Response 200 POST /session/370fe688bdb4bc37e3c0514f4418a4a3/elements (16ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '10746a0c-62f9-45b2-9464-bff810997d66'
       }
     ]
  }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (19ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (6ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (9ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (7ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (7ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms)
   { value: 'Example Domain' }
   Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text  

   Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms)
   { value: 'Example Domain' }
  ✖ Expected element <body > div > h1> text to contain: "blah" - expected "contain 'blah'" but got: "Example Domain" (5152ms)
    at DescribeInstance.<anonymous> (/Users/cjackson/Documents/nwrepro/tests/reproAfterEach.js:9:20) 

  → Completed command: expect.element ('body > div > h1') (5153ms)
  TimeoutError: done() callback timeout of 20000ms was reached while executing "afterEach". Make sure to call the done() callback when the operation finishes.
   
 → Running [after]:
 → Completed [after].
 
 → Running command: end ()
 
 → Running command: session ('delete', [Function])
   Request DELETE /session/370fe688bdb4bc37e3c0514f4418a4a3  

   Response 200 DELETE /session/370fe688bdb4bc37e3c0514f4418a4a3 (53ms)
   { value: null }
  → Completed command: session ('delete', [Function]) (54ms)
 Wrote log file to: /Users/cjackson/Documents/nwrepro/logs/reproAfterEach_chromedriver.log.
  → Completed command: end () (57ms)

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  TEST FAILURE (21.57s): 
   - 1 error during execution; 
   - 1 assertions failed; 0 passed



 TimeoutError: done() callback timeout of 20000ms was reached while executing "afterEach". Make sure to call the done() callback when the operation finishes.
   

 ✖ 1) reproAfterEach
 – fail on purpose 
   Expected element <body > div > h1> text to contain: "blah" - expected "contain 'blah'" but got: "Example Domain" (5152ms)
       at DescribeInstance.<anonymous> (/Users/cjackson/Documents/nwrepro/tests/reproAfterEach.js:9:20)

 ChromeDriver process closed.
 Wrote report file to: tests_output/CHROME_101.0.4951.64__reproAfterEach.xml.

Configuration

nightwatch.json

//
// Refer to the online docs for more details:
// https://nightwatchjs.org/gettingstarted/configuration/
//
//  _   _  _         _      _                     _          _
// | \ | |(_)       | |    | |                   | |        | |
// |  \| | _   __ _ | |__  | |_ __      __  __ _ | |_   ___ | |__
// | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \
// | |\  || || (_| || | | || |_  \ V  V / | (_| || |_ | (__ | | | |
// \_| \_/|_| \__, ||_| |_| \__|  \_/\_/   \__,_| \__| \___||_| |_|
//             __/ |
//            |___/
//

module.exports = {
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['tests'],

  // See https://nightwatchjs.org/guide/working-with-page-objects/using-page-objects.html
  page_objects_path: ['node_modules/nightwatch/examples/pages/'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/custom-commands.html
  custom_commands_path: ['node_modules/nightwatch/examples/custom-commands/'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/custom-assertions.html
  custom_assertions_path: '',

  // See https://nightwatchjs.org/guide/extending-nightwatch/plugin-api.html
  plugins: [],

  // See https://nightwatchjs.org/guide/#external-globals
  globals_path: '',

  webdriver: {},

  test_settings: {
    default: {
      disable_error_log: false,
      launch_url: 'https://nightwatchjs.org',

      screenshots: {
        enabled: false,
        path: 'screens',
        on_failure: true,
      },

      desiredCapabilities: {
        browserName: 'firefox',
      },

      webdriver: {
        start_process: true,
        server_path: '',
      },
    },

    safari: {
      desiredCapabilities: {
        browserName: 'safari',
        alwaysMatch: {
          acceptInsecureCerts: false,
        },
      },
      webdriver: {
        start_process: true,
        server_path: '',
      },
    },

    firefox: {
      desiredCapabilities: {
        browserName: 'firefox',
        alwaysMatch: {
          acceptInsecureCerts: true,
          'moz:firefoxOptions': {
            args: [
              // '-headless',
              // '-verbose'
            ],
          },
        },
      },
      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // very verbose geckodriver logs
          // '-vv'
        ],
      },
    },

    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ],
        },
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ],
      },
    },

    edge: {
      desiredCapabilities: {
        browserName: 'MicrosoftEdge',
        'ms:edgeOptions': {
          w3c: true,
          // More info on EdgeDriver: https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/capabilities-edge-options
          args: [
            //'--headless'
          ],
        },
      },

      webdriver: {
        start_process: true,
        // Download msedgedriver from https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/
        //  and set the location below:
        server_path: '',
        cli_args: [
          // --verbose
        ],
      },
    },

    //////////////////////////////////////////////////////////////////////////////////
    // Configuration for when using cucumber-js (https://cucumber.io)                |
    //                                                                               |
    // It uses the bundled examples inside the nightwatch examples folder; feel free |
    // to adapt this to your own project needs                                       |
    //////////////////////////////////////////////////////////////////////////////////
    'cucumber-js': {
      src_folders: ['examples/cucumber-js/features/step_definitions'],

      test_runner: {
        // set cucumber as the runner
        type: 'cucumber',

        // define cucumber specific options
        options: {
          //set the feature path
          feature_path:
            'node_modules/nightwatch/examples/cucumber-js/*/*.feature',

          // start the webdriver session automatically (enabled by default)
          // auto_start_session: true

          // use parallel execution in Cucumber
          // parallel: 2 // set number of workers to use (can also be defined in the cli as --parallel 2
        },
      },
    },

    //////////////////////////////////////////////////////////////////////////////////
    // Configuration for when using the browserstack.com cloud service               |
    //                                                                               |
    // Please set the username and access key by setting the environment variables:  |
    // - BROWSERSTACK_USERNAME                                                       |
    // - BROWSERSTACK_ACCESS_KEY                                                     |
    // .env files are supported                                                      |
    //////////////////////////////////////////////////////////////////////////////////
    browserstack: {
      selenium: {
        host: 'hub.browserstack.com',
        port: 443,
      },
      // More info on configuring capabilities can be found on:
      // https://www.browserstack.com/automate/capabilities?tag=selenium-4
      desiredCapabilities: {
        'bstack:options': {
          userName: '${BROWSERSTACK_USERNAME}',
          accessKey: '${BROWSERSTACK_ACCESS_KEY}',
        },
      },

      disable_error_log: true,
      webdriver: {
        timeout_options: {
          timeout: 15000,
          retry_attempts: 3,
        },
        keep_alive: true,
        start_process: false,
      },
    },

    'browserstack.local': {
      extends: 'browserstack',
      desiredCapabilities: {
        'browserstack.local': true,
      },
    },

    'browserstack.chrome': {
      extends: 'browserstack',
      desiredCapabilities: {
        browserName: 'chrome',
        chromeOptions: {
          w3c: true,
        },
      },
    },

    'browserstack.firefox': {
      extends: 'browserstack',
      desiredCapabilities: {
        browserName: 'firefox',
      },
    },

    'browserstack.ie': {
      extends: 'browserstack',
      desiredCapabilities: {
        browserName: 'internet explorer',
        browserVersion: '11.0',
      },
    },

    'browserstack.safari': {
      extends: 'browserstack',
      desiredCapabilities: {
        browserName: 'safari',
      },
    },

    'browserstack.local_chrome': {
      extends: 'browserstack.local',
      desiredCapabilities: {
        browserName: 'chrome',
      },
    },

    'browserstack.local_firefox': {
      extends: 'browserstack.local',
      desiredCapabilities: {
        browserName: 'firefox',
      },
    },
    //////////////////////////////////////////////////////////////////////////////////
    // Configuration for when using the SauceLabs cloud service                      |
    //                                                                               |
    // Please set the username and access key by setting the environment variables:  |
    // - SAUCE_USERNAME                                                              |
    // - SAUCE_ACCESS_KEY                                                            |
    //////////////////////////////////////////////////////////////////////////////////
    saucelabs: {
      selenium: {
        host: 'ondemand.saucelabs.com',
        port: 443,
      },
      // More info on configuring capabilities can be found on:
      // https://wiki.saucelabs.com/display/DOCS/Test+Configuration+Options
      desiredCapabilities: {
        'sauce:options': {
          username: '${SAUCE_USERNAME}',
          accessKey: '${SAUCE_ACCESS_KEY}',
          // https://docs.saucelabs.com/dev/cli/sauce-connect-proxy/#--region
          // region: 'us-west-1'
          // https://docs.saucelabs.com/dev/test-configuration-options/#tunnelidentifier
          // parentTunnel: '',
          // tunnelIdentifier: '',
        },
      },
      disable_error_log: false,
      webdriver: {
        start_process: false,
      },
    },
    'saucelabs.chrome': {
      extends: 'saucelabs',
      desiredCapabilities: {
        browserName: 'chrome',
        screenResolution: '1280x1024',
        browserVersion: 'latest',
        javascriptEnabled: true,
        acceptSslCerts: true,
        timeZone: 'London',
        chromeOptions: {
          w3c: true,
        },
      },
    },
    'saucelabs.firefox': {
      extends: 'saucelabs',
      desiredCapabilities: {
        browserName: 'firefox',
        screenResolution: '1280x1024',
        browserVersion: 'latest',
        javascriptEnabled: true,
        acceptSslCerts: true,
        timeZone: 'London',
      },
    },
    //////////////////////////////////////////////////////////////////////////////////
    // Configuration for when using the Selenium service, either locally or remote,  |
    //  like Selenium Grid                                                           |
    //////////////////////////////////////////////////////////////////////////////////
    selenium_server: {
      // Selenium Server is running locally and is managed by Nightwatch
      // Install the NPM package @nightwatch/selenium-server or download the selenium server jar file from https://github.com/SeleniumHQ/selenium/releases/, e.g.: selenium-server-4.1.1.jar
      selenium: {
        start_process: true,
        port: 4444,
        server_path: '', // Leave empty if @nightwatch/selenium-server is installed
        command: 'standalone', // Selenium 4 only
        cli_args: {
          //'webdriver.gecko.driver': '',
          //'webdriver.chrome.driver': ''
        },
      },
      webdriver: {
        start_process: false,
        default_path_prefix: '/wd/hub',
      },
    },

    'selenium.chrome': {
      extends: 'selenium_server',
      desiredCapabilities: {
        browserName: 'chrome',
        chromeOptions: {
          w3c: true,
        },
      },
    },

    'selenium.firefox': {
      extends: 'selenium_server',
      desiredCapabilities: {
        browserName: 'firefox',
        'moz:firefoxOptions': {
          args: [
            // '-headless',
            // '-verbose'
          ],
        },
      },
    },
  },
};

Your Environment

Executable	Version
nightwatch --version	2.15
npm --version	8.1.2
yarn --version	1.22.17
node --version	v16.13.1

Browser driver	Version
chromedriver	101.0.4951.41

OS	Version
macOS Big Sur	11.6.5

[AutomatedTester]

To be tested with https://github.com/nightwatchjs/nightwatch/pull/3328

[gravityvi]

Removing async will fix the issue as I can see it's not required. You can either return a promise or use await both will resolve the issue.

  it('fail on purpose',  async function() {
    browser.url('http://example.com/');
    await browser.expect.element('body > div > h1').text.to.contain('blah');
  });

  it('fail on purpose', async function() {
    browser.url('http://example.com/');
    return browser.expect.element('body > div > h1').text.to.contain('blah');
  });