playwright
playwright copied to clipboard
microsoft
[Regression]: Can't call test.use() inside a describe() suite of a different test
Last Good Version
1.41.2
First Bad Version
1.42.0
Steps to reproduce
After updating to playwright 1.42, I get this message about one of my test files which was fine in 1.41.2:
Can't call test.use() inside a describe() suite of a different test type.
Make sure to use the same "test" function (created by the test.extend() call) for all declarations inside a suite.playwright
To reproduce, just have describe blocks with extend/use inside of them.
import { test as base } from '@playwright/test';
base.describe('Admin user', () => {
// Create an admin item
const test = base.extend({ item: async ({}, use) => {} });
/***** Here is the line that triggers errors *****/
test.use({ storageState: '/admin/user/path' });
// Tests with admin item
test('1', async ({ page, item }) => {});
test('2', async ({ page, item }) => {});
});
base.describe('Normal user', () => {
// Create a normal item
const test = base.extend({ item: async ({}, use) => {} });
/***** Here is the line that triggers errors *****/
test.use({ storageState: '/normal/user/path' });
// Tests with normal item
test('1', async ({ page, item }) => {});
test('2', async ({ page, item }) => {});
});
Expected behavior
I expect to be allowed to have use/extend inside a describe block.
Actual behavior
I am not allowed to have use/extend inside a describe block.
Additional context
No response
Environment
System:
OS: macOS 14.3.1
CPU: (10) arm64 Apple M1 Pro
Memory: 3.52 GB / 32.00 GB
Binaries:
Node: 20.11.1 - ~/.nvm/versions/node/v20.11.1/bin/node
Yarn: 1.22.21 - /opt/homebrew/bin/yarn
npm: 10.2.4 - ~/.nvm/versions/node/v20.11.1/bin/npm
IDEs:
VSCode: 1.86.2 - /opt/homebrew/bin/code
Languages:
Bash: 3.2.57 - /bin/bash
npmPackages:
@playwright/test: ^1.42.0 => 1.42.0
Allowing the following code was an oversight on our side. Mixing the test instances (and hooks called on those instances) in the same suite prevents us from keeping the strict semantic.
const test = base.extend({ item: async ({}, use) => {} });
base.describe('Admin user', () => {
test('1', async ({ page, item }) => {});
test('2', async ({ page, item }) => {});
});
I'll leave this change open to see how many people are affected by this.
I updated the release notes to include this as a breaking change. Is there a reason you need to mix test instances? Can you fix the code?
Also affected here. I've popped a question on the Discord group but cross posting for visibility:
I have the following setup for visual testing:
describe('goes to confirm page', () => {
beforeEach(async ({ page }) => {
await formToConfirmation(page);
await form.titlePage.waitFor();
});
test('@visual desktop', async ({ captureScreenshot }) => {
await captureScreenshot('aml-contacts-desktop-confirm');
});
testMobile('@visual mobile', async ({ captureScreenshot }) => {
await captureScreenshot('aml-contacts-mobile-confirm');
});
});
testMobile comes from our playwright.config.ts file:
export const testMobile = test.extend<PlaywrightTestArgs>({
viewport: { width: 414, height: 896 },
});
We've found this a nice way to manage our tests, however is the expectation now to have a separate describe block when the test instances change? It's a shame as this seemed to offer quite a maintainable approach to our testing at scale.
I updated the release notes to include this as a breaking change. Is there a reason you need to mix test instances? Can you fix the code?
@pavelfeldman There is no particular reason, I just thought it looked neat the way I wrote it. Since it wasn't mentioned in the release notes I wasn't sure if it was an intentional change or not.
I can also do this, it was only to avoid having fixtures like adminItem and normalItem without splitting into seperate files, so I nested the fixtures to have them both be item.
(If it is still wrong, since I have different test.use per describe block, please let me know.)
import { test as base } from '@playwright/test';
const test = base.extend({
adminItem: async ({}, use) => {},
normalItem: async ({}, use) => {},
});
test.describe('Admin user', () => {
test.use({ storageState: '/admin/user/path' });
test('1', async ({ page, adminItem }) => {});
test('2', async ({ page, adminItem }) => {});
});
test.describe('Normal user', () => {
test.use({ storageState: '/normal/user/path' });
test('1', async ({ page, normalItem}) => {});
test('2', async ({ page, normalItem }) => {});
});
We are experiencing difficulties due to this recent update, and we'd like to share our code snippet to highlight its repercussions. Adapting to this change requires significant effort, as it's disrupting the functionality of nearly all our tests.
The error we get is
Error: Can't call test() inside a describe() suite of a different test type. Make sure to use the same "test" function (created by the test.extend() call) for all declarations inside a suite.
Our setup involves categorizing shows based on user access or the presence of a paywall, or sometimes both. This feature is critical not only for our organization but also for every other organization using Playwright. As others begin upgrading to the latest version, the absence of this feature will undoubtedly have a widespread impact on test organization for all users. Therefore, we kindly request the restoration of this feature to ensure smooth operations for all users.
import freeContentShow from 'path/to/freeContentMultipleSeasons.json';
import { expect, test } from 'path/to/test';
const { platform } = config;
const fixtures = {
free: ['show1', 'show2', 'Season 5'],
premium: ['showA', 'showB', 'Season 2']
};
const [freeShowUrl, premiumShowUrl, seasonName] = fixtures[platform];
test.describe('Test Description 1', () => {
test.beforeEach(async ({ page, platform }) => {
await page.route(`**/catalog/v2/${platform}/show/${freeShowUrl}*`, async (route) => {
const json = freeContentShow[platform];
await route.fulfill({ json });
});
});
test('Test Name 1', async ({ page, platform }) => {
await page.goto(`/${freeShowUrl}`);
await expect(page.getByLabel(seasonName)).toHaveScreenshot(`free-season-${platform}.png`);
});
});
test.describe('Test Description 2', () => {
test.beforeEach(async ({ page, platform }) => {
await page.route(`**/catalog/v2/${platform}/show/${premiumShowUrl}*`, async (route) => {
const json = require(`path/to/showAllMediaPremium.json`);
await route.fulfill({ json });
});
});
test('Test Name 2', async ({ page, platform }) => {
await page.goto(`/${premiumShowUrl}`);
await expect(page.getByLabel(seasonName)).toHaveScreenshot(`premium-season-${platform}.png`);
});
});
I am also having issues with this change. I was using this to set up an electron app or browser within the same test suite.
We are affected by this change as well.
In our case we have test groupings which, after consuming dynamically the current combination under test, determines a particular version of the test group to be executed.
We're affected by this change as well. we are using under same describe default "test" and modified "test" fixtures to create test groups
@pavelfeldman Any updates as to whether this breaking change will be reverted? Our playwright testing framework is built around this functionality, and we would very much like to maintain the current way that it is working, At the moment, we're stuck on version 1.41, and we're in a bit of uncertainty as to what the future holds (whether we will have to change our architectural approach completely or whether we should wait to see if this breaking change will be reverted).
Yes, same issue Error: Can't call test.describe() inside a describe() suite of a different test type Out automation test using playwright were using two structures of myTest and test, and now it's not working. any suggestions on this? I have to rewire all to separate each?
not sure if it is related or not, but we're on version 1.40
And getting this error:
Cannot use({ fixture1 }) in a describe group, because it forces a new worker.
Make it top-level in the test file or put in the configuration file.
where we try to do something like the code below. We use fixtures quite heavily and they set up a lot of our entities, but lately we've come to scenarios where we would potentially not want them invoked conditionally per test (so can't use a global setting such as auto: true/false.
test.describe(
'hello world',
() => {
if(condition) test.use({ fixture1: '' });
test.only( 'just testing', async ({ fixture2, fixture1 }) => {
// ...do some things
})
}
)
Also affected by this, we have put off upgrading playwright until a decision is made here
I don't like having to duplicate code
const test = base.extend({ ... })
const authTest = test.extend({ ... })
// Before
test.describe('desc', () => {
test.beforeEach(() => { ... })
test('test', () => { ... })
authTest('test', () => { ... })
})
// After
test.describe('desc', () => {
test.beforeEach(() => { ... })
test('test', () => { ... })
})
authTest.describe('desc (authenticated)', () => {
authTest.beforeEach(() => { duplicate code })
authTest('test', () => { ... })
})
Probably not ideal but my workaround is to create a fixture that I load strictly for its side effects whenever I want to modify the page context:
const test = base.extend<{ __AUTH__: Page }>({
__AUTH__: async({ page }, use) => {
await page.addInitScript(() => {})
await use(page)
},
})
test.describe('desc', () => {
test.beforeEach(() => { ... })
test('test', () => { ... })
test('test', ({ __AUTH__ }) => { await __AUTH__.reload(); ... })
})
Personally I think any extension of a test should be allowed to be nested instead of it. For example:
const A = base.extend({ ... })
const A1 = A.extend({ ... })
const B = base.extend({ ... })
A.describe('', () => {
A.test('', () => { }) // works as expected
A1.test('', () => { }) // should be allowed
B.test('', () => { }) // should not be allowed (?)
})
Yeah, to me it seems that this is a breaking change and hence shouldn't be included in a minor version regardless of reasoning.
Mixing the test instances (and hooks called on those instances) in the same suite prevents us from keeping the strict semantic.
@pavelfeldman could you please explain what exactly you mean here? Maybe there's another solution.
Can we get an update on this? I'm not able to update @playwright/test to the latest because of this breaking change. Is it going to be reverted, or should we start refactoring all of our tests for this? @pavelfeldman
This change also affects a project I am working on. +1 It would be really really great if it's fixed in the next release version.
