[BUG] Command line `--grep-invert` option overrides project-level `grepInvert` configuration

[ozyx]

Context:

• Playwright Version: 1.25.2

System:

• OS: macOS 12.5.1
• Memory: 31.85 MB / 16.00 GB

Binaries:

• Node: 16.16.0 - ~/.nvm/versions/node/v16.16.0/bin/node
• npm: 8.11.0 - ~/.nvm/versions/node/v16.16.0/bin/npm

Languages:

• Bash: 3.2.57 - /bin/bash

Describe the bug

We are running into an issue where the --grep-invert option on the command line is overriding the grepInvert for our individual project configurations.

Our use case is as such:

• We are executing our nightly test run, which runs against all of our project configurations.
• We have multiple projects, but we only want to run tests annotated with @snapshot on one of them. So we have configured all projects except one to have grepInvert: /@snapshot/
• We have tests annotated with @couchdb which we don't want to run as part of our full nightly test run. So we have added the --grep-invert='@couchdb' command line option to our npm script for triggering a full nightly run.

But it seems that the command line option is overriding the individual project configurations as @snapshot tests are being run by all projects. One solution which would work for us is if the regexes can be compounded together as if it were an 'AND'. If this is not possible, it would be nice to have this behavior documented so it's clear you should use one or the other.

I have a repro repo located here: https://github.com/ozyx/grepInvert-repro Run npm run test to see the behavior.

[rwoll]

Thanks for the report (and your repo)! I will raise with the team to discuss tomorrow. I've boiled it down to:

Given:

import type { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  projects: [
    { name: 'my-project', grepInvert: /@snapshot/ }
  ],
};

export default config;

and

import { test } from '@playwright/test';

test('should work with images @snapshot', () => {});
test('should work withdb @couchdb', () => {});

This output looks expected:

$ npx playwright test --project=my-project --list
Listing tests:
  [my-project] › tests/example.spec.ts:4:1 › should work withdb @couchdb
Total: 1 test in 1 file

but this (adding --grep-invert '@couchdb') is not expected:

$ npx playwright test --project=my-project --grep-invert '@couchdb' --list
Listing tests:
  [my-project] › tests/example.spec.ts:3:1 › should work with images @snapshot
Total: 1 test in 1 file

The expectation is with --grep-invert '@couchdb', the above should match no tests.

[ozyx]

The expectation is with --grep-invert '@couchdb', the above should match no tests.

Yup! That's correct. Thanks, and let me know if you need any other info.

[rwoll]

As a workaround for now, you can manually union your filters on the CLI, e.g.:

npx playwright test --project=my-project --grep-invert '@couchdb|@snapshot' --list

There's some upcoming changes relating to grouping tests, and that API will be additive.

[unlikelyzero]

@yury-s @rwoll it's worth noting that we are implementing this to specifically skip the execution of snapshot tests. The recent change to https://github.com/microsoft/playwright/releases/tag/v1.26.0 for actually provides the native capability we were shimming into tags