🔧 Repository: Run basic playwright tests as part of CI workflow

[jrichter1]

📜 Description

Some plugins/workspaces have playwright tests that check basic functionality using dev pages. It would be nice if these workspaces could declare the desire to run such tests as part of the CI workflow.

A workspace could add something like

"playwrightTests": true,

to its bcp.json file.

In the CI workflow, a new step would be added that would install playwright with dependencies, and simply run it with the worspace as cwd:

yarn playwright install --with-deps chromium chrome // the most likely browsers to be used
yarn playwright test

This would require the workspace to contain a valid playwright.config.ts file in its root. Any other test dependencies will be the responsibility of the workspace as well.

👍 Expected behavior

This is not a bug, but there was no repository feature request template

👎 Current Behavior

This is not a bug, but there was no repository feature request template

👟 Reproduction steps

This is not a bug, but there was no repository feature request template

📃 Provide the context for the Bug.

No response

👀 Have you spent some time to check if this bug has been raised before?

• [x] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [x] I have read the Code of Conduct

Are you willing to submit PR?

Yes I am willing to submit a PR!

[awanlin]

Hi @jrichter1, conceptually this makes sense to me as something people can opt into. But is this a current reality? Like are any of the current test anything more than boiler plate? Also, we'll likely need some way to handle/mock the related 3rd party service most of the plugins in this repo require to work. 🤔

[jrichter1]

I've seen some UI tests in plugins like RBAC that actually seem to be doing something meaningful.

As far as mocking goes, each plugin can handle it as they desire. Supplying data to their dev pages, or directly mocking the requests within playwright.

If the convention would be, say, the workspace has a playwright config, that config defines a webServer which would be nothing but the plugin doing something like yarn start. Calling yarn playwright test would launch the dev environment, and the tests afterwards. That way it's all isolated to the workspace, and any mocks are their responsibility.

[christoph-jerolimov]

It would be really great to start allowing plugins / workspaces to run playwright tests directly on GitHub (workflows).

And yep, how useful these tests are or if they have external dependencies or if the requires a big set of mocks is of course up to the plugin. Some dev apps might be able to test easily without a backend, but it should be possible to use the backend as well. For example with such a command in the workspace package.json:

    "dev": "yarn workspaces foreach -A --include @backstage-community/plugin-<plugin> --include @backstage-community/plugin-<plugin>-backend --parallel -v -i run start",

Or even better, directly in the playwright.config.ts. The backstage scaffolder creates something like this which might need an update for the CI case then... :D

  // Run your local dev server before starting the tests
  webServer: process.env.CI
    ? []
    : [
        {
          command: 'yarn start app',
          port: 3000,
          reuseExistingServer: true,
          timeout: 60_000,
        },
        {
          command: 'yarn start backend',
          port: 7007,
          reuseExistingServer: true,
          timeout: 60_000,
        },
      ],

So actually this is what I do after each backstage update as a first step. Run the dev pages and see if they are still working.

I think an opt-in feature is a great first step that doesn't affect any workspace that doesn't use this. And of course, we could extend this to new workspace in the scaffolder content or see if we can have a simple test for more plugins later.

The only question I have currently is: Are we expected that these tests lives in the workspace or will some plugins authors might have these tests in their plugin? Maybe it doesn't matter, right? The Backstage scaffolder app (default playwright.config.ts) uses this again to find their tests:

  projects: generateProjects(), // Find all packages with e2e-test folders

This means the e2e-tests folders can live in ~~any package or~~ any plugin folder, right?

I would like to see a PR for this and I like to help to enabling tests for example for my npm package.

[jrichter1]

I've opened #6274 to show how this could work.

Using quay as an example, all the necessary configuration is in its playwright config file.

It's launching the dev page for testing, the data is supplied directly to the dev page. The CI simply installs playwright dependencies and launches the test according to the config.

[HusneShabbir]

It might be a good idea to split this into two steps for better clarity and maintainability:

• Separation of Concerns: Installing dependencies and running tests could be distinct actions, which makes it easier to debug and isolate issues.
• Cleaner Output: Having separate steps might give you clearer logs, making it easier to spot whether issues are in the installation or testing phase.

This could improve control and simplify the CI process overall.