[Feature]: A hook to run before `webServer`

[kettanaito]

🚀 Feature Request

Hi!

It would be great if Playwright supported an optional hook that would allow developers to execute custom logic related to their tests before Playwright spawns the tested application (i.e. executes webServer.command).

With that, I am proposing two new hooks:

interface ProjectConfig {
  runnerSetup?: string
  runnerTeardown?: string
}

Mirroring the globalSetup and globalTeardown hooks, the newly proposed ones allow you to include a setup/teardown file and await that respective logic before spawning your tested app.

With the proposed hooks, here's the order of things:

- runnerSetup
  - webServer
  - globalSetup
    - beforeAll
    - afterAll
  - globalTeardown
- runnerTeardown

Example

Basic usage

The runnerSetup and runnerTeardown hooks behave identically to globalSetup and globalTeardown, in a sense that they expect the specified modules to have a default export describing the logic to run:

// runner-setup.ts
export default async function runnerSetup() {
  setupLogic()
}

// runner-teardown.ts
export default async function runnerTeardown() {
  teardownLogic()
}

// playwright.config.ts
export default defineConfig({
  runnerSetup: './runner-setup.ts',
  runnerTeardown: './runner-teardown.ts',
})

Shorthand teardown declaration

Similar to the globalSetup optionally returning the teardown callback, I propose the runnerSetup support the same pattern:

export default async runnerSetup({ context }) {
  const container = await new PostgresContainer().start()

  return async () => {
    await container.stop()
  }
}

This is extremely handy to help manage persisted resources, like my testcontainer above.

Motivation

Currently, there's no way to run custom logic before the webServer step of Playwright:

• webServer is a static object, not a function to allow logic (primarily async one) to run before spawning the app (and spawning the app dynamically based on that logic).
• globalSetup runs after webServer, making it impossible to influence the spawned app (e.g. by providing different env based on some async logic). This is intended as you expect your tested app to be up by this point to create browsers and do custom browser/page-related setup.

Because of this, it becomes redundantly complex to introduce trivial custom logic related to the test run. Take the aforementioned Testcontainers as an example. Without the proposed hooks, I need to:

• Introduce a custom script.
• Start the testcontainer there.
• Spawn the entire Playwright process dynamically to provide dynamic env variables.
• Manage the spawned PW process.
• Close the testcontainer manually.

This is an extreme level of complexity for something as small as spawning something alongside but before my tested app. I strongly suggest you consider this proposal to simplify such setups as I believe they are rather common.

Any other approaches, like chaining shell scripts or creating a custom webServer.command, suffer from the same issue: they introduce redundant complexity because you are trying to manage a test runner-related resource outside of that test runner.

[dgozman]

@kettanaito Thank you for the issue. Have you tried prepending an extra webServer command into the config.webServer array? This would run before the actual web server.

[kettanaito]

@dgozman, yes, that's one solution I considered, but as I mentioned in the proposal, that introduces a lot of complexity to manage as now I'm splitting the test setup into two separate processes.

I believe that my side effect is directly a part of the test setup and should be managed close to the test runner. Effectively, what I'm proposing is globalSetup that runs before the app so webServer can be more dynamic.

[dgozman]

@kettanaito Could you help me understand how runnerSetup: './myscript.js' is different from an entry in webServer that says command: 'node ./myscript.js'? We consider these two to be basically identical.

[kettanaito]

Of course!

The main difference between runnerSetup and any kind of manual logic orchestration is the connection to the test runner's life cycle. With the designated hook, Playwright becomes responsible for calling it at the appropriate time during the test run. In our case, before the webServer runs.

With manual shell scripts, that responsibility shifts toward the developer. That is not to say they cannot manage that, only that they shouldn't, really. Consider the DX difference between these two approaches to solve the same problem:

Approach 1: Designated test runner hook

// runner-setup.ts
export default async function runnerSetup() {
  const container = await new TestContainer().start()
  process.env.DB_URL = container.getConnectionUri()

  return () => contaoner.stop()
}

export default defineConfig({
  runnerSetup: './runner-setup.ts',
  webServer: {
    command: 'npm start',
    env: {
      DB_URL: process.env.DB_URL
    }
  }
})

Approach 2: Custom scripts

// run-playwright.ts
import { spawnSync } from 'node:child_process'

async function runTests() {
  const container = await new TestContainer()
  await container.start()

  const io = await spawnSync('npx', ['playwright', 'test'], {
    env: {
      ...process.env,
      DB_URL: container.getConnectionUri()
    }
  })

  io.on('exit', async () => {
    await container.stop()
  })
}

playwright.config.ts remains largely the same here, save for the runnerSetup, naturally.

Conclusion

Notice how one approach feels like configuring your test runner to have the test setup you need, and another one feels like creating a custom test orchestration pipeline.

The latter approach introduced a significant volume of complexity. Your test setup becomes dependent not on the test runner itself but on the custom process where you spawn that runner. A lot of developers never even touched spawnSync but they might need to spawn a test container to have a controlled DB instance for their E2E tests in Playwright. The complexity of the use case is disproportional to the complexity of the solution.

That's not even touching on the fact that the solution above is a huge workaround. You have to manually implement CLI options forwarding to the underlying Playwright process. Oops, I forgot to forward stdio at all, so I won't see my test output at all. This isn't the level of customization I'm comfortable recommending to anyone.

[dgozman]

@kettanaito Thank you for the explanation. I understand why you don't like the wrapper script approach. However, I was suggesting something entirely different. In the example below we have a very similar runner-setup file that is used as an additional web server in the list of web servers. Let me know what you think.

// runner-setup.mjs
const container = await new TestContainer().start()

// Assuming ".env" is loaded by the actual webserver.
await fs.promises.writeFile('.env', `DB_URL=${container.getConnectionUri()}`)

// In rare cases where you need some custom cleanup logic.
// I assume that most things will just stop upon process termination.
process.on('SIGTERM', async () => {
  await container.stop()
  process.exit(0)
})

// playwright.config.ts
export default defineConfig({
  webServer: [
    {
      command: 'node ./runner-setup.mjs',
      // Include the next line if you have custom SIGTERM handling.
      gracefulShutdown: { signal: 'SIGTERM', timeout: 5000 },
    },
    {
      command: 'npm start',
      // You can read and pass DB_URL here if web server does not do it.
    }
  ],
})

[kettanaito]

I've considered multiple web servers but wrote off that approach since it wasn't mentioned anywhere that Playwright would run those servers sequentially. Is that the case? Does Playwright guarantee that webServer[0] finishes spawning before webServer[1] starts? Otherwise, the first web server may not yet set the env variable when the second one, the one dependent on it, will start running (and thus be broken).

Another question I have here is how to tell Playwright when your runner-setup.mjs is "ready"? The docs state that either port or url option must be provided on the web server. What if I cannot provide either? What if I want to spawn my test container on a random port, which is both the default behavior and the best practice when it comes to test-related resources. I was under the impression that the webServer properties weren't suitable for this kind of scenarios.

[dgozman]

@kettanaito We have discussed this topic with the team. We'd really like to make webServer as versatile as possible before introducing new APIs, so below are some suggestions. Please let us know whether something like that would work for you.

What do you think about having a readyMessage option for the webServer? This would be used instead of port or url, and Playwright will wait for this message to be printed in stdout/stderr before it considers your web server ready. This way, you can print something once your test container is ready.

Additionally, we can wait for the webServer command to just finish. This would work if you start some service or do one-off preparation work, and don't need to hold a process to cleanup.

Regarding the sequential webServer execution, Playwright guarantees it today, and I don't see it changing in the future.

[kettanaito]

I fully support the notion of exploring how to use the existing before introducing the new.

What do you think about having a readyMessage option for the webServer?

If I read this right, this option only affects when Playwright thinks my app is ready. If I actually want to make sure that logicA runs before npm start, readyMessage won't help here. Playwright still fires webServer.command and is not in charge of the dependencies that happen there (rightfully so). I'm not sure the readyMessage option solve my use case.

On a related note, if you are considering an std-driven predicate, and that predicate is a new option anyway, why not something like waitUntil() that is a function. This is unrelated to what we are discussing, rather two cents from me:

{
  webServer: {
    waitUntil({ stdout, stderr }) {
      return stdout.includes('Application is running!')
    }
  }
}

Regarding the sequential webServer execution, Playwright guarantees it today, and I don't see it changing in the future.

Okay, this is the most important bit. With this in mind, I can see where you're coming from suggesting readyMessage. If I can do something like this, I believe it would work for me:

{
  webServer: [
    {
      command: 'npm run test:db',
      waitUntil() {
        return process.env.DB_URL != null
      }
    },
    {
      command: 'npm start',
      port: 3000,
      env: {
        // Illustrational.
        DB_URL: process.env.DB_URL
      }
    }
  ]
}

Notice how a generic waitUntil is far more versatile than asking the developer to print something to stdout. In my case, I only care that the setup step sets the correct environment variable. Once it does, I consider that logic to have finished running.

But here I have a question: does Playwright run multiple webServer as forks of the same parent process? I wonder if I even can set an env variable from one web server to then be read in my playwright.config.ts?

Additionally, we can wait for the webServer command to just finish.

Not sure how good of an idea it is to introduce this dichotomy of the behavior. Conventionally, webServer is meant to run an app, and the app process is not meant to finish (i.e. exit). That'd mean a broken setup. Introducing another behavior that tolerates command exits means now Playwright is responsible for figuring out which exits are intended vs which are not. I wouldn't do that. I like and support the idea of an eventual predicate (port, url, readyMessage, or waitUntil).

The new API

I strongly encourage you to consider a function-based predicate for webServer. Here are my reasons why.

1. It's infinitely more versatile, which means it likely covers virtually any use case.
2. You can rewrite the existing port and url predicates to use waitUntil under the hood, thus minimizing the functionality to maintain.
3. Ideally, Playwright shouldn't be the judge of when the developer setup logic is considered complete. As in, assuming it prints something somewhere is a broad assumption. Instead, let the developer describe when their logic is complete as they are responsible for it.

[dgozman]

But here I have a question: does Playwright run multiple webServer as forks of the same parent process? I wonder if I even can set an env variable from one web server to then be read in my playwright.config.ts?

I don't think so, we spawn a separate process with its own env. My example from this comment suggests a file instead.

On a related note, if you are considering an std-driven predicate, and that predicate is a new option anyway, why not something like waitUntil() that is a function

Do you have any other examples on how you'd use a function? Given that webServer spawns a process, there's not too many ways to communicate with it except for stdio, tcp or files. So we'd cover two out three with readyMessage. I am asking because we prefer declarative APIs over imperative, especially so in the config file. In fact, we'd love our config to be entirely json-serializable.

[kettanaito]

I don't think so, we spawn a separate process with its own env.

Which means even if I spawned my custom setup logic, I still need to implement more logic in order for it to pass data to another webServer? Hm, that's not good. Precisely the complexity bump I'm trying to avoid with this proposal.

Relying on a file, or another side effect, means you need to manage that now. .env.test isn't that bad, but still for it to work not Playwright has to add yet another option, this one checking for a file's existence. Would that be acceptable with the direction of the config you try to maintain?

Do you have any other examples on how you'd use a function?

• process.env[foo] was set;
• A file exists in the file system;
• A port becomes occupied;
• A request to a URL returns 200 (or a custom response);

These are the examples I can think of. If you wish to have a serializable config, then I suppose they translate to

• ???
• file
• port
• url

I'm not a big fan of these names as they aren't descriptive enough (webServer.port is ambiguous because it doesn't belong to webServer, it's a predicate; as in, might as well mean this is where I instruct Playwright to spawn the app).

[dgozman]

@kettanaito We've discussed this one a bit more.

It seems like we do not fully agree with the approach where you orchestrate your setup in the test runner, instead of outside of test runner. We do have webServer that does exactly that, but it feels more like a compromise for a very popular request.

Ideally, we would like users to setup their environment, including any containers or services, separately before running tests. This makes it more flexible in terms of CI vs local dev, and usually easier to manage with some scripts or npm commands instead of trying to squeeze that into the test runner.

In your example, we probably expect to see something like that:

# This one starts any necessary containers/services and writes into .env
- npm run start-containers

# This one starts the server, reading .env if needed
- npm run start-server

# Finally run tests
- npm run e2e

Thank you for filing the issue! I'll leave it open to collect upvotes and discuss ideas.

[kettanaito]

@dgozman, appreciate your feedback and leaving it open. What you're suggesting is certainly possible and applicable for test setup. If I have any more thoughts, I will share them in this thread. It might take me a while as I'm not actively working on the project I need this for, but I would love to find an elegant solution to suggest to my students.

[kettanaito]

A quick confirmation on my end: all entries in the webServer array run in parallel. Which means you cannot use them to orchestrate your setup (e.g. migrate your DB, then spawn your app). This is likely for the best as your test setup doesn't belong in your test runner's config.

This means that any app-related setup you need in your e2e tests cannot be done on Playwright level at all. It must be done by orchestrating processes, I suppose. Then, arranging IPC of some sort to have them communicate info, like the DATABASE_URL to use (think Testcontainers + Playwright).

I really wish Playwright considered providing developers a way to establish test setup before running the tested app. Fingers crossed.