Playwright tests for Next 13 app using app router fail unless slowMo set fairly high

[AmyBlankenship]

trafficstars

Verify canary release

• [X] I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: win32
      Arch: x64
      Version: Windows 10 Home
    Binaries:
      Node: 16.14.2
      npm: N/A
      Yarn: N/A
      pnpm: N/A
    Relevant Packages:
      next: 13.4.11-canary.0
      eslint-config-next: N/A
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 4.9.5
    Next.js Config:
      output: N/A

Which area(s) of Next.js are affected? (leave empty if unsure)

No response

Link to the code that reproduces this issue or a replay of the bug

https://github.com/AmyBlankenship/next-contacts-example/tree/feat/add-address

To Reproduce

• Clone branch above
• yarn install
• npm run test:e2e

Note: if the tests fail, you need to run posttest:e2e manually

Describe the Bug

With the project as committed, the "adding address" test fails in chrome, firefox, and edge (note I disabled safari early on because I was getting failures only in safari and figured it was bc this is a windows machine--not sure if this is relevant).

If you add:

launchOptions: { slowMo: 1000, },

below line 35 of playwright.config, now chrome and edge pass.

If you change that to launchOptions: { slowMo: 2000, },

all three browsers pass.

Expected Behavior

Should not need to set playwright to slowMo to make tests pass (and if that's necessary, it should be clearly documented). Note that I don't know how to see what the browser versions are in playwright, but I just installed the latest as part of trying to figure this out.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

[matejtarca]

Any news here? Currently experiencing the same issue with next 13.5.4 and playwright 1.39.0. If the slowMo is not specified, the page suddenly changes in the middle of a test to a completely different one (happening in somewhat random places but usually after a mutation). I think it's because of the revalidation of the path in server actions 🤔

EDIT: I found a workaround by introducing a manual wait in Playwright right after accessing a new page. That way a global slowMo is not needed, and the tests pass. I want to point out that I am testing on a standalone production build, not a dev server, so I am unsure why this would be an issue.

[AmyBlankenship]

I'm not working in Next at my current job, but thanks for the update

[sjconstantin]

I am seeing this issue also. Am am testing for this

await expect(this.page.getByText('Logged in as:')).toBeVisible();

This passes when testing against my app running in development mode. Running in production mode the test fails without slowMo. If I set slowMo to 2000 the test passes in production mode.

[pieterjanscheir]

Does anyone have any update or other workarounds for this issue?

[manavm1990]

I didn't know about slowMo until I came across this 🧵 . However, I have gone from using geByLabel to getByLocator and now there are at least fewer flake issues.

diff --git a/apps/web/e2e/auth.spec.ts b/apps/web/e2e/auth.spec.ts
index d7ca2fc..9f3a5a3 100644
--- a/apps/web/e2e/auth.spec.ts
+++ b/apps/web/e2e/auth.spec.ts
@@ -14,9 +14,7 @@ test("Login and out", async ({ page }) => {
   expect(process.env.TEST_PASSWORD).toBeDefined();
 
   await page.getByLabel("Email ✉️").fill(process.env.TEST_USERNAME!);
-  await page
-    .getByLabel("Password", { exact: true })
-    .fill(process.env.TEST_PASSWORD!);
+  await page.locator('input[name="password"]').fill(process.env.TEST_PASSWORD!);
 
   const loginButton = page.getByRole("button", { name: "Log In" });
   await loginButton.click();
@@ -37,7 +35,7 @@ test("Validation", async ({ page }) => {
   await usernameInput.fill(process.env.TEST_USERNAME!);
   await expect(alert).not.toBeVisible();
 
-  const passwordInput = page.getByLabel("Password", { exact: true });
+  const passwordInput = page.locator('input[name="password"]');
   await passwordInput.fill("a");
   await expect(alert).toContainText("Must be 8 or more characters long");
   await passwordInput.fill(process.env.TEST_PASSWORD!);

[jameslporter]

Have a next 14 app dir app and cannot get playwright to work hardly at all while running in dev mode. Switch to build & start and it works fine, passes every time and takes 40s +-4s. Overall dev mode is frustratingly slow and prod mode is crazy fast. Have some issues with turbo mode so I'm not using that. Using bun runtime and it seems stable just uses more memory than node did.