[Feature] make existing trace option in the top-level use block apply to globalSetup & include in report

[stephenkilbourn]

It is possible to manually set up tracing for global config. However, it would make debugging simpler if we make the existing trace option in the top-level use block apply to globalSetup. We also need a way to include this trace in the report .

Perhaps there can be an initial Setup section after the .subnav-search and then the test output sections

Related to #14889

[rwoll]

Summary from offline discussion:

• many folks request tracing for globalSetup (and for it to show up in the HTML Report)
• there's already trace-related options in config for the tests, not for globalSetup
• we don't love how you have to manually configure traces in globalSetup
• we don't love how it's not clear which browser you should/need use in globalSetup
• it's not clear how the trace settings in the config would apply to globalSetup if we wanted to wire them up automagically
• it's not clear where we would attach trace for globalSetup (globalInfo.atttachemnts, maybe?); where do we show them in the report

Proposal:

Add a special "init" test(s) that are authored and configured like any other test (including per-project), but always run before all the other tests.

• can be retried
• respects tracing options from config
• displayed in the HTML report like any other test
• fixtures provided making it easier to author

[TimeInvestor]

Looking forward to this feature 😊

[rwoll]

See https://github.com/microsoft/playwright/issues/14895 documenting some of the motivation for this proposal. Also see this GH query or skim through Slack to find numerous occurrences of users asking how to debug and trace globalSetup (which is now documented here).

The goal of this proposal is to make it easier to run global-setup-like code in an environment that's pre-configured with the existing Playwright Config, and also have a clearer debug/reporting story.

I'm still experimenting around, but documenting some ideas to discuss before implementing. These are partial notes.

Straw Man Proposed API

testInit specifies the path to a JavaScript or TypeScript file that contains exactly one test.init block:

Example Config

module.exports = {
  testInit: "./pre-auth-user.init.ts",
  use: {
    storageState: "./default-storage-state.json",
  },
  projects: [
    { name: "mobile", use: { ...devices["iPhone 11"] }, testInit: "./mobile-pre-auth.init.ts", use: { storageState: "./mobile-storage-state.json" } },
    { name: "desktop chrome", … },
    { name: "desktop safari", … },
    { name: "desktop firefox", … },
  ]
}

Example Test Init

test.use({ storageState: undefined });

test.init("pre-auth user", async ({ config, playwright, browser, context, page, useStorageState }) => {
  await page.goto("https://example.com/login");
  await page.locator("text=Username").fill(process.env.TEST_USERNAME);
  await page.locator("text=Password").fill(process.env.TEST_PASSWORD);
  await page.locator("text=Sign In").click();
  await expect(page.locator("body")).toContainText("Successfully authenticated!", { useInnerText: true });

  useStorgaeState(browser);
  useEnv({ … }); // gross: just capture process.env
  
// await context.saveStorageState({ path: config.use.storageState });
});

• test.init behaves simlarly to a regular test(…) case
• all options in the config (like retries) apply to it by default[^1]
• If at least one project does not specify a testInit override, the top-level testInit is run at least once.
• project-level testInits are run at least once.
• testInit runs after the configuration phase, but prior to any other regular tests running.
• if retries are enabled in the config (via the current retry settings), it will be run until it works once within the allotted retries
• in reports, test.init runs show up like current tests, but have an init annotation (much like how fixme/skip behave)
• the top-level browser's BrowserType does not necessarily align with a project
• config is a test.init specific fixture, since the test.init will need to know where to save storage state
• .init.ts suffix is used so it's not picked up by the searches for *.spec.ts; although, we do not automatically discover the init.ts file. It must be explicitly configured.
• test.init runs in a random worker
• test.init ends BEFORE any other tests are run, so you can't have anything persistent in mem or set env vars

---

• env issue is a blocker

• storageState API

• test.deinit

• how do we pass ports e.g. if you start a server and want other tests to use it?

• dima likes explicitly setting init file

• going a step further:

schedule: [
  { tag: '@init', mode: 'serial' },
  { tag: '*' },
  { tag: '@cleanup', mode: 'serial' }
]

Keeping a worker and it's in-memory state around:

schedule: [
  { tag: '@init', worker: "abc" },
  { tag: '*'},
  { tag: '@cleanup', worker: "abc" }
]

^ This would be like invoking playwright test three times.

Alternatively, instead of tags, use filenames.

• What happens if worker needs to be torn down?
• How does beforeAll fit into this? Does it run before the @init?
• This looks a lot like specifying projects, but just specifying order.

test('aaa', { test$init } => {})

---

fixture('aaa', { page, context }, use)

[^1]: Maybe—what do we do about something like storageState?

[vijaydubber]

Hi @rwoll , I have a few questions on https://github.com/microsoft/playwright/issues/14895#issuecomment-1203272743

1. when we say module.exports = {}, will it actually become part of PlaywrightTestConfig (hoping so)

const config: PlaywrightTestConfig = {
testInit: "./pre-auth-user.init.ts",
}

2. can we also have a teardown

const config: PlaywrightTestConfig = {
testInit: "./pre-auth-user.init.ts",
testTeardown: "./pre-auth-user.teardown.ts",
}

or auto-look for teardown in init
const config: PlaywrightTestConfig = {
testInit: "./pre-auth-user.init.ts",
testTeardown: "./pre-auth-user.init.ts",
}

and
pre-auth-user.init.ts
test.init('i am init')
test.teardown('i am teardown')

3. in debug mode: can we ensure PW debugger also launches when are in init / teardown blocks. Currently it doesn't.

[rwoll]

Folded #16215 into this where a user requests that their fixtures are available in globalSetup.

[rwoll]

Here's an updated proposal based on conversations around https://github.com/microsoft/playwright/issues/14895#issuecomment-1203272743. It's inspired by @pavelfeldman's simpler schedule proposal, but takes it a step further to accommodate more use cases requested by users.

We do not introduce test.init.

Instead, we introduce a schedule API which allows the user to specify a function that receives the entire test suite/cases and returns a dependency tree of test cases which is used to influence execution order. It looks similar to a Custom Reporter's onBegin signature. This is not a coincidence. We also provide some common schedule functions for the most common use cases.

In the Playwright config, at the top-level of the config, the user can supply a function of type Scheduler:

type Scheduler = (config: ReadOnly<FullConfig>, suite: Suite) => TestDeps;
type TestDeps = { dependsOn: TestDeps, testCases: TestCase[] };

Suite and and the types found within its data structure provide project, Location/filename, title, etc., so this allows for all sorts of different scheduling strategies. For example:

1. Setup + Teardown: Run a test titled "setup" before all other tests that generates auth state to be consumed by all other tests (and run a test titled "teardown" after all other tests).
2. Sequential Projects: Do not interleave running multiple project's tests. Run project A, before project B, then run project C.
3. Ordered Files: Run certain test files before others.

This proposal satisfies[^1]:

1. common request for setup/teardown (and their traces) to appear in HTML Report—everything is just a test (https://github.com/microsoft/playwright/issues/14895)
2. apply tracing settings from config to the setup/teardown (https://github.com/microsoft/playwright/issues/14895)
3. access fixtures in setup/teardown (https://github.com/microsoft/playwright/issues/16215)
4. file ordering (https://github.com/microsoft/playwright/issues/16232)

Some additional notes:

• with and without TestDeps returned by the user, tests have implicit ordering rules based on fixtures that will still be respected
• to keep the config serializable, we might want to avoid an inline scheduler function and instead opt to use a string filename that points to a file with the function (and/or names one of the common strategies).
• setting/sharing env vars across workers: we'll introduce a new built-in fixture captureEnv which, when called captures process.env and those values are used as process.env in all dependent tests
• .only will implicitly run all tests leading to it

[^1]: I link to some current issues, but it shall be noted we've received these requests numerous times in various forms over various channels.

[rwoll]

Feedback:

• show examples of the basic use cases
• remove dependsOn/tree struct — just return an array of arrays. Each level depends on the last.
• don't expose the full API, just expose the common schedulers (i.e. the 3 documented above)

[rwoll]

Simplified API proposal to implement:

Experimented with some more complicated scheduling logic, but ultimately a nice first step would be to make it simpler, review actual impl, and poke holes in it.

Proposed API to implement (and review impl with team):

(Naming arbitrary at the moment.)

Introduce top-level config projectSchedule:

projectSchedule: [
    ["global setup"],
    ["mobile setup", "desktop setup"],
    ["mobile"],
    ["desktop"],
    ["global teardown"]
]

The names refer to actual projects defined in the config, we run all projects within a row of the array (in an arbitrary order) prior to running the projects in the next row of the array.

[camsteffen]

Would it make sense to have scope: 'global' for fixtures? As in, a global fixture runs before workers are spawned.

[Khivar]

Simplified API proposal to implement:

Experimented with some more complicated scheduling logic, but ultimately a nice first step would be to make it simpler, review actual impl, and poke holes in it.

Proposed API to implement (and review impl with team):

(Naming arbitrary at the moment.)

Introduce top-level config projectSchedule:

projectSchedule: [
    ["global setup"],
    ["mobile setup", "desktop setup"],
    ["mobile"],
    ["desktop"],
    ["global teardown"]
]

The names refer to actual projects defined in the config, we run all projects within a row of the array (in an arbitrary order) prior to running the projects in the next row of the array.

Saw you made a proof of concept pull request at the end of August for this feature but closed it due to a new API proposal that was discussed offline. Anywhere we can read more on this ?

[carlpaten]

We have test suites with nested dependencies, e.g. I can't test cluster creation if sign-up doesn't work, and I can't test cluster usage if cluster creation doesn't work (or if cluster creation couldn't run because sign-up doesn't work). This could be handled by .serial() but we want to run in parallel the different branches of the dependency tree: testing account administration relies on sign-up, but it doesn't depend on cluster creation, so the two should be able to run in parallel.

So we need some kind of DAG construct for tests. Ideally this DAG would be respected when using playwright -g to run a subset of all tests at a time, but that's just a convenience, not a dealbreaker if we don't have it. The programming interface for this could be as simple as providing a list of dependencies when declaring a test.

Since this isn't available at the moment I'm guessing that there must be some solution I haven't thought of that everybody else is using for parallelization of complex and interdependent tests. Any tips are welcome.

[pavelfeldman]

Starting with v1.31 you'll be able to shape your global setup as a Playwright project: https://playwright.dev/docs/next/api/class-testproject#test-project-dependencies. You can use top-level test.use block in the setup files that way.

[neg3ntropy]

I have been trying v1.31 and configured things as per @pavelfeldman's comment above, but I cannot seem to get my desired behavior that is to execute a db cleanup step once and only once before the whole test run. Not once per worker, but once globally and even in the case that I am selecting a single test file or grepping.

[dgozman]

@neg3ntropy Could you share a repro that illustrates the issue, so that we can run it locally?

[yury-s]

Closing per https://github.com/microsoft/playwright/issues/14895#issuecomment-1414388361 Please file a new issue if it doesn't work for you.