nightwatch
nightwatch copied to clipboard
Screenshots are not taken on failure if mixing async test with sync afterEach test hook
Description of the bug/issue
When I run my test with screenshots enabled on failure and my test fails I expect a screenshot to be saved under the screenshots directory of the current browser screen, but this is not happening when the test is async and I am using a non-async afterEach hook.
Steps to reproduce
- Create an async test that will fail
- Add an afterEach hook that is not async
- Run the test. No screenshot is saved.
Sample test
import { NightwatchBrowser } from 'nightwatch';
module.exports = {
'should take a screenshot': async (browser: NightwatchBrowser) => {
browser.assert.fail('oops');
},
afterEach: (browser: NightwatchBrowser) => {
browser.navigateTo('https://www.ecosia.org/');
},
};
Command to run
npx nightwatch .\nightwatch
Verbose Output
npx nightwatch .\nightwatch --verbose
Now you can run TS tests directly using Nightwatch.
Launching up to 1 concurrent test worker processes...
Running: default: test2.ts
DevTools listening on ws://127.0.0.1:56766/devtools/browser/9448f3e0-9483-4ad3-9f50-69ddb5c04fc0
× default: test2.ts Now you can run TS tests directly using Nightwatch.
[Nightwatch\test2] Test Suite
──────────────────────────────────────────────────────────
Starting ChromeDriver with server_path=C:\Users\Mr\Desktop\screenshotTest\node_modules\chromedriver\lib\chromedriver\chromedriver.exe...
Request POST /session
{
capabilities: {
firstMatch: [ {} ],
alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} }
}
}
Response 200 POST /session (633ms)
{
value: {
capabilities: {
acceptInsecureCerts: false,
browserName: 'chrome',
browserVersion: '113.0.5672.127',
chrome: {
chromedriverVersion: '113.0.5672.63 (0e1a4471d5ae5bf128b1bd8f4d627c8cbd55f70c-refs/branch-heads/5672@{#912})',
userDataDir: 'C:\\Users\\Mr\\AppData\\Local\\Temp\\scoped_dir13232_1753912520'
},
'goog:chromeOptions': { debuggerAddress: 'localhost:56766' },
networkConnectionEnabled: false,
pageLoadStrategy: 'normal',
platformName: 'windows',
proxy: {},
setWindowRect: true,
strictFileInteractability: false,
timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
unhandledPromptBehavior: 'dismiss and notify',
'webauthn:extension:credBlob': true,
'webauthn:extension:largeBlob': true,
'webauthn:extension:minPinLength': true,
'webauthn:extension:prf': true,
'webauthn:virtualAuthenticators': true
},
sessionId: 'e7922c34d80c125a32fdca1e4207430c'
}
}
Using: chrome (113.0.5672.127) on WINDOWS.
Received session with ID: e7922c34d80c125a32fdca1e4207430c
→ Running [before]:
→ Completed [before].
– should take a screenshot
→ Running [beforeEach]:
→ Completed [beforeEach].
→ Running [afterEach]:
→ Running command: assert.fail ()
✖ NightwatchAssertError
oops
Error location:
C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:
––––––––––––––––––––––––––––––––––––––––––––––––––––––––
3 | module.exports = {
4 | 'should take a screenshot': async (browser: NightwatchBrowser) => {
5 | browser.assert.fail('oops');
6 | },
7 |
––––––––––––––––––––––––––––––––––––––––––––––––––––––––
→ Completed command: assert.fail () (4ms)
→ Completed [afterEach].
× default: test2.ts [Nightwatch\test2] should take a screenshot (37ms)
oops
at Object.should take a screenshot (C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:5:20)
→ Running [after]:
→ Completed [after].
→ Running command: end (true)
→ Running command: session ('delete', [Function])
Request DELETE /session/e7922c34d80c125a32fdca1e4207430c
Response 200 DELETE /session/e7922c34d80c125a32fdca1e4207430c (88ms)
{ value: null }
→ Completed command: end (true) (108ms)
→ Completed command: session ('delete', [Function]) (92ms)
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
️TEST FAILURE (3.425s):
- 1 assertions failed; 0 passed
× 1) nightwatch\test2
– should take a screenshot (37ms)
→ ✖ NightwatchAssertError
oops
Error location:
C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:
––––––––––––––––––––––––––––––––––––––––––––––––––––––––
3 | module.exports = {
4 | 'should take a screenshot': async (browser: NightwatchBrowser) => {
5 | browser.assert.fail('oops');
6 | },
7 |
––––––––––––––––––––––––––––––––––––––––––––––––––––––––
Wrote HTML report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch-html-report\index.html
Wrote JSON report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch\CHROME_113.0.5672.127__test2.json
Wrote Rerun Json report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\minimal_report.json
Wrote XML report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch\CHROME_113.0.5672.127__test2.xml
Nightwatch Configuration
// Refer to the online docs for more details:
// https://nightwatchjs.org/gettingstarted/configuration/
//
// _ _ _ _ _ _ _
// | \ | |(_) | | | | | | | |
// | \| | _ __ _ | |__ | |_ __ __ __ _ | |_ ___ | |__
// | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \
// | |\ || || (_| || | | || |_ \ V V / | (_| || |_ | (__ | | | |
// \_| \_/|_| \__, ||_| |_| \__| \_/\_/ \__,_| \__| \___||_| |_|
// __/ |
// |___/
module.exports = {
// An array of folders (excluding subfolders) where your tests are located;
// if this is not specified, the test source must be passed as the second argument to the test runner.
src_folders: ['test', 'nightwatch'],
// See https://nightwatchjs.org/guide/concepts/page-object-model.html
page_objects_path: [],
// See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
custom_commands_path: [],
// See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
custom_assertions_path: [],
// See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
plugins: [],
// See https://nightwatchjs.org/guide/concepts/test-globals.html
globals_path: '',
webdriver: {},
test_workers: {
enabled: true,
},
test_settings: {
default: {
disable_error_log: false,
launch_url: 'http://localhost',
screenshots: {
enabled: true,
path: 'screens',
on_failure: true,
},
desiredCapabilities: {
browserName: 'chrome',
},
webdriver: {
start_process: true,
server_path: '',
},
},
chrome: {
desiredCapabilities: {
browserName: 'chrome',
'goog:chromeOptions': {
// More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
//
// w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
w3c: true,
args: [
//'--no-sandbox',
//'--ignore-certificate-errors',
//'--allow-insecure-localhost',
//'--headless'
],
},
},
webdriver: {
start_process: true,
server_path: '',
cli_args: [
// --verbose
],
},
},
},
};
Nightwatch.js Version
2.6.21
Node Version
18.16.0
Browser
Chrome 113
Operating System
Windows 10
Additional Information
If the afterEach test hook is marked async the screenshot is saved, but it seems like it should work in either case
❌ afterEach: (browser: NightwatchBrowser) => {
✅ afterEach: async (browser: NightwatchBrowser) => {
In your sample test you don't have an await. Could you retry the following snippet. Your async
doesn't have an await
.
import { NightwatchBrowser } from 'nightwatch';
module.exports = {
'should take a screenshot': async (browser: NightwatchBrowser) => {
await browser.assert.fail('oops');
},
afterEach: (browser: NightwatchBrowser) => {
browser.navigateTo('https://www.ecosia.org/');
},
};
So, maybe a silly question, but if the test is marked async does it require an await? browser.assert doesn't typically need to be awaited so now I'm confused.
Preferably, yes. We can't detect if you're using await
so the test will continue without any warning and in most cases I think it will be fine, but in some situations, like yours the next testcase or the afterEach hook will start sooner than expected, so the queue might get confused.
Even though in theory the internal queue should be able to handle these situations, it's best to try and avoid them.
So in async test cases is the recommendation to await every line or ensure at least one await is used?
Yes, await every line, but you can still use chaining, you don't have to await every single command, you can await a chain as well.
I'm running into a similar issue on nightwatch 3.5.0
where screenshots aren't being taken on failure.
Sample test:
import { NightwatchAPI } from 'nightwatch'
import { Google } from '../../page-objects'
describe('Google', function () {
this.tags = ['google', 'ui']
let google: Google
before(function (browser: NightwatchAPI) {
google = browser.page.google()
})
beforeEach(function() {
google.navigate()
})
afterEach(function (browser: NightwatchAPI) {
browser.end()
})
it('image should be visible', async function () {
await browser.assert.fail('oops')
})
})
I tried making the afterEach
async and await browser.end()
and that didn't work. Tried using the done
callback and that didn't work either. Tried using arrow functions but that didn't work.
The only thing that did work was removing async from the test (it
) but I would expect it to work with a async
test function as well. It seems the like the issue is around having the test be marked as async.
Any ideas what could be happening or any recommendations? I can open a new issue if needed. I would prefer using async/await
so I don't do too many nested call backs.