nightwatch icon indicating copy to clipboard operation
nightwatch copied to clipboard

Screenshots are not taken on failure if mixing async test with sync afterEach test hook

Open reallymello opened this issue 1 year ago • 6 comments

Description of the bug/issue

When I run my test with screenshots enabled on failure and my test fails I expect a screenshot to be saved under the screenshots directory of the current browser screen, but this is not happening when the test is async and I am using a non-async afterEach hook.

Steps to reproduce

  1. Create an async test that will fail
  2. Add an afterEach hook that is not async
  3. Run the test. No screenshot is saved.

Sample test

import { NightwatchBrowser } from 'nightwatch';

module.exports = {
  'should take a screenshot': async (browser: NightwatchBrowser) => {
    browser.assert.fail('oops');
  },

  afterEach: (browser: NightwatchBrowser) => {
    browser.navigateTo('https://www.ecosia.org/');
  },
};

Command to run

npx nightwatch .\nightwatch

Verbose Output

npx nightwatch .\nightwatch --verbose
 Now you can run TS tests directly using Nightwatch.
 Launching up to 1 concurrent test worker processes...

 Running:  default: test2.ts 

DevTools listening on ws://127.0.0.1:56766/devtools/browser/9448f3e0-9483-4ad3-9f50-69ddb5c04fc0


×  default: test2.ts   Now you can run TS tests directly using Nightwatch. 
  
 [Nightwatch\test2] Test Suite 
 ────────────────────────────────────────────────────────── 
 Starting ChromeDriver with server_path=C:\Users\Mr\Desktop\screenshotTest\node_modules\chromedriver\lib\chromedriver\chromedriver.exe... 
   Request POST /session   
 { 
      capabilities: { 
        firstMatch: [ {} ], 
        alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} } 
      } 
   } 
   Response 200 POST /session (633ms) 
 { 
      value: { 
        capabilities: { 
          acceptInsecureCerts: false, 
          browserName: 'chrome', 
          browserVersion: '113.0.5672.127', 
          chrome: { 
            chromedriverVersion: '113.0.5672.63 (0e1a4471d5ae5bf128b1bd8f4d627c8cbd55f70c-refs/branch-heads/5672@{#912})', 
            userDataDir: 'C:\\Users\\Mr\\AppData\\Local\\Temp\\scoped_dir13232_1753912520' 
          }, 
          'goog:chromeOptions': { debuggerAddress: 'localhost:56766' }, 
          networkConnectionEnabled: false,
          pageLoadStrategy: 'normal',
          platformName: 'windows',
          proxy: {},
          setWindowRect: true,
          strictFileInteractability: false,
          timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
          unhandledPromptBehavior: 'dismiss and notify',
          'webauthn:extension:credBlob': true,
          'webauthn:extension:largeBlob': true,
          'webauthn:extension:minPinLength': true,
          'webauthn:extension:prf': true,
          'webauthn:virtualAuthenticators': true
        },
        sessionId: 'e7922c34d80c125a32fdca1e4207430c'
      }
   }
 Using: chrome (113.0.5672.127) on WINDOWS.
 Received session with ID: e7922c34d80c125a32fdca1e4207430c 
  
 → Running [before]:
 → Completed [before].

 – should take a screenshot
 → Running [beforeEach]:
 → Completed [beforeEach].
 → Running [afterEach]:
  
  → Running command: assert.fail () 
  ✖ NightwatchAssertError
    oops
  
     Error location: 
     C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:
     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––
      3 | module.exports = {
      4 |   'should take a screenshot': async (browser: NightwatchBrowser) => {
      5 |     browser.assert.fail('oops'); 
      6 |   },
      7 |
     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––

  → Completed command: assert.fail () (4ms)
 → Completed [afterEach].
 × default: test2.ts [Nightwatch\test2] should take a screenshot (37ms)
    oops
        at Object.should take a screenshot (C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:5:20)
 → Running [after]:
 → Completed [after].
  
  → Running command: end (true) 
  
  → Running command: session ('delete', [Function]) 
   Request DELETE /session/e7922c34d80c125a32fdca1e4207430c  

   Response 200 DELETE /session/e7922c34d80c125a32fdca1e4207430c (88ms)
 { value: null }
  → Completed command: end (true) (108ms)
  → Completed command: session ('delete', [Function]) (92ms)


───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (3.425s):
   - 1 assertions failed; 0 passed

   × 1) nightwatch\test2

   – should take a screenshot (37ms)

   → ✖ NightwatchAssertError
   oops

    Error location:
    C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     3 | module.exports = {
     4 |   'should take a screenshot': async (browser: NightwatchBrowser) => {
     5 |     browser.assert.fail('oops'); 
     6 |   },
     7 |
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––


 Wrote HTML report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch-html-report\index.html

 Wrote JSON report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch\CHROME_113.0.5672.127__test2.json
 Wrote Rerun Json report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\minimal_report.json
 Wrote XML report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch\CHROME_113.0.5672.127__test2.xml

Nightwatch Configuration

// Refer to the online docs for more details:
// https://nightwatchjs.org/gettingstarted/configuration/
//

//  _   _  _         _      _                     _          _
// | \ | |(_)       | |    | |                   | |        | |
// |  \| | _   __ _ | |__  | |_ __      __  __ _ | |_   ___ | |__
// | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \
// | |\  || || (_| || | | || |_  \ V  V / | (_| || |_ | (__ | | | |
// \_| \_/|_| \__, ||_| |_| \__|  \_/\_/   \__,_| \__| \___||_| |_|
//             __/ |
//            |___/

module.exports = {
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['test', 'nightwatch'],

  // See https://nightwatchjs.org/guide/concepts/page-object-model.html
  page_objects_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
  custom_commands_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
  custom_assertions_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
  plugins: [],

  // See https://nightwatchjs.org/guide/concepts/test-globals.html
  globals_path: '',

  webdriver: {},

  test_workers: {
    enabled: true,
  },

  test_settings: {
    default: {
      disable_error_log: false,
      launch_url: 'http://localhost',

      screenshots: {
        enabled: true,
        path: 'screens',
        on_failure: true,
      },

      desiredCapabilities: {
        browserName: 'chrome',
      },

      webdriver: {
        start_process: true,
        server_path: '',
      },
    },

    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ],
        },
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ],
      },
    },
  },
};

Nightwatch.js Version

2.6.21

Node Version

18.16.0

Browser

Chrome 113

Operating System

Windows 10

Additional Information

If the afterEach test hook is marked async the screenshot is saved, but it seems like it should work in either case

❌ afterEach: (browser: NightwatchBrowser) => {

✅ afterEach: async (browser: NightwatchBrowser) => {

reallymello avatar May 30 '23 14:05 reallymello

In your sample test you don't have an await. Could you retry the following snippet. Your async doesn't have an await.

import { NightwatchBrowser } from 'nightwatch';

module.exports = {
  'should take a screenshot': async (browser: NightwatchBrowser) => {
    await browser.assert.fail('oops');
  },

  afterEach: (browser: NightwatchBrowser) => {
    browser.navigateTo('https://www.ecosia.org/');
  },
};

AutomatedTester avatar Jun 13 '23 10:06 AutomatedTester

So, maybe a silly question, but if the test is marked async does it require an await? browser.assert doesn't typically need to be awaited so now I'm confused.

reallymello avatar Jun 13 '23 11:06 reallymello

Preferably, yes. We can't detect if you're using await so the test will continue without any warning and in most cases I think it will be fine, but in some situations, like yours the next testcase or the afterEach hook will start sooner than expected, so the queue might get confused.

Even though in theory the internal queue should be able to handle these situations, it's best to try and avoid them.

beatfactor avatar Jun 14 '23 12:06 beatfactor

So in async test cases is the recommendation to await every line or ensure at least one await is used?

reallymello avatar Jun 14 '23 12:06 reallymello

Yes, await every line, but you can still use chaining, you don't have to await every single command, you can await a chain as well.

beatfactor avatar Jun 15 '23 09:06 beatfactor

I'm running into a similar issue on nightwatch 3.5.0 where screenshots aren't being taken on failure.

Sample test:

import { NightwatchAPI } from 'nightwatch'
import { Google } from '../../page-objects'

describe('Google', function () {
  this.tags = ['google', 'ui']
  let google: Google

  before(function (browser: NightwatchAPI) {
    google = browser.page.google()
  })

  beforeEach(function() {
    google.navigate()
  })

  afterEach(function (browser: NightwatchAPI) {
    browser.end()
  })

  it('image should be visible', async function () {
    await browser.assert.fail('oops')
  })
})

I tried making the afterEach async and await browser.end() and that didn't work. Tried using the done callback and that didn't work either. Tried using arrow functions but that didn't work.

The only thing that did work was removing async from the test (it) but I would expect it to work with a async test function as well. It seems the like the issue is around having the test be marked as async.

Any ideas what could be happening or any recommendations? I can open a new issue if needed. I would prefer using async/await so I don't do too many nested call backs.

simonwang384 avatar Mar 29 '24 15:03 simonwang384