studio

studio copied to clipboard

Quick summary

The E2E tests are currently failing, both locally and in CI.

I think the reason this hasn't been noticed in PRs is that the workflow step is configured with continue-on-error: true.

Steps to reproduce

Locally, run npm install && npm run package && npm run e2e.

On CI, visit a recent PR and look at its actions. For example, #56: https://github.com/Automattic/studio/actions/runs/8830011430/job/24242246592?pr=27

What you expected to happen

E2E tests should be able to create tmp sites and move forward with the testing.

What actually happened

Locally, you can see the app being stuck on the "Add your first site" screen:

I assume the same happens in CI, based on the way the tests fail, but I have no way to verify it.

Impact

One

Available workarounds?

There is no user impact

Platform

No response

Logs or notes

Here's my local test log

➜ npm run e2e


> [email protected] e2e
> npx playwright install && npx playwright test


Running 4 tests using 1 worker

  ✓  1 app.test.ts:18:6 › Electron app › should ensure app title is correct. (242ms)
  ✘  2 sites.test.ts:34:6 › Servers › create a new site (15.6s)
  ✘  3 sites.test.ts:60:6 › Servers › edit site's settings in wp-admin (505ms)
  ✘  4 sites.test.ts:77:6 › Servers › delete site (5.5s)


  1) sites.test.ts:34:6 › Servers › create a new site ──────────────────────────────────────────────

    Error: locator.inputValue: Error: Node is not an <input>, <textarea> or <select> element
    Call log:
      - waiting for getByRole('dialog', { name: 'Add a site' }).getByLabel('Local path')


      38 | 		await modal.siteNameInput.fill( siteName );
      39 | 		await modal.clickLocalPathButtonAndSelectFromEnv();
    > 40 | 		expect( await modal.localPathInput.inputValue() ).toBe( tmpSiteDir );
         | 		                                   ^
      41 | 		await modal.addSiteButton.click();
      42 |
      43 | 		const sidebarButton = sidebar.getSiteNavButton( siteName );

        at /Users/gio/Developer/a8c/studio/e2e/sites.test.ts:40:38

  2) sites.test.ts:60:6 › Servers › edit site's settings in wp-admin ───────────────────────────────

    Error: page.goto: net::ERR_CONNECTION_REFUSED at http://localhost:8881/wp-admin/options-general.php
    Call log:
      - navigating to "http://localhost:8881/wp-admin/options-general.php", waiting until "load"


      66 |
      67 | 		// page.goto opens a browser
    > 68 | 		await page.goto( wpAdminUrl + '/options-general.php' );
         | 		           ^
      69 | 		const siteTitleInput = page.getByLabel( 'Site Title' );
      70 | 		await siteTitleInput.fill( 'testing site title' );
      71 | 		await siteTitleInput.press( 'Enter' );

        at /Users/gio/Developer/a8c/studio/e2e/sites.test.ts:68:14

  3) sites.test.ts:77:6 › Servers › delete site ────────────────────────────────────────────────────

    Error: Timed out 5000ms waiting for expect(locator).toBeVisible()

    Locator: getByRole('dialog', { name: 'Delete test-site-1ad71ca0-c394-4059-ae58-59195465f3af' })
    Expected: visible
    Received: hidden
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for getByRole('dialog', { name: 'Delete test-site-1ad71ca0-c394-4059-ae58-59195465f3af' })


       at page-objects/settings-tab.ts:43

      41 | 		await this.deleteButton.click();
      42 | 		const modal = new DeleteSiteModal( this.page, this.siteName );
    > 43 | 		await expect( modal.locator ).toBeVisible();
         | 		                              ^
      44 | 		return modal;
      45 | 	}
      46 | }

        at SettingsTab.openDeleteSiteModal (/Users/gio/Developer/a8c/studio/e2e/page-objects/settings-tab.ts:43:33)
        at /Users/gio/Developer/a8c/studio/e2e/sites.test.ts:81:17

  Slow test file: sites.test.ts (21.6s)
  Consider splitting slow test files to speed up parallel execution
  3 failed
    sites.test.ts:34:6 › Servers › create a new site ───────────────────────────────────────────────
    sites.test.ts:60:6 › Servers › edit site's settings in wp-admin ────────────────────────────────
    sites.test.ts:77:6 › Servers › delete site ─────────────────────────────────────────────────────
  1 passed (29.7s)

I updated #38 to upload the Playwright traces.

They can be downloaded individually and inspected with

npx playwright show-trace ~/Downloads/trace.zip

Here's what the one "create a new site" shows:

Two things to notice:

• The console shows that the element exists, but it's detached from the DOM. I'm not sure what this means practically, given that the code should return the Onboarding component if no site is present, and that's what one experiences locally as per my screenshot above.
• There doesn't seem to be any view in the trace. I don't know why that's the case, but that's also what happens when running `npm run e2e -- --ui

I have a crude fix for the onboarding issue at mokagio/e2e-fix-attempt. The branch is a WIP mess of experiments and comments, so I don't recommend using it. Just leaving a breadcrumb in case I get sidetracked from other tasks.

I'll take a look at the failures on Windows (https://github.com/Automattic/studio/pull/76).

I reviewed the current state of this issue. E2e tests are going through on local and build servers, but only for Mac. We still need to fix unit and e2e tests on Windows and enable them for the PR builds.

I reviewed the current state of this issue. E2e tests are going through on local and build servers, but only for Mac. We still need to fix unit and e2e tests on Windows and enable them for the PR builds.

Yep. I tried to address the issues on Windows some time ago unsuccessfully (https://github.com/Automattic/studio/compare/trunk...test/windows-e2e-tests). For some reason, the test runner seems to get stuck when creating the first site. I couldn't find any clue to narrow down the culprit, but I wonder if it has to do with the OS. Maybe a system prompt that is waiting for user response and blocks the E2E tests. In any case, I couldn't confirm the latter but would be interesting to investigate further.

I reviewed the current state of this issue. E2e tests are going through on local and build servers, but only for Mac. We still need to fix unit and e2e tests on Windows and enable them for the PR builds.

Thinking twice, we could consider closing the issue as E2E tests are only failing on Windows. This could be tracked in a separate issue. WDYT?

Thinking twice, we could consider closing the issue as E2E tests are only failing on Windows. This could be tracked in a separate issue. WDYT?

@fluiddot makes sense, I will close this one and we can continue with the new issue.