[Feature] For `toHaveScreenshot` report: align failure report file names with snapshot filename

[mscottford]

Goal

Improve the developer experience when manually "approving" screenshots verified with toHaveScreenshot. This could also be a stepping stone towards adding UI to "approve" differences.

Background

Images generated from running the tests are stored in the test-results directory using the following naming convention.

./test-results/<dashed base spec file name>-<dashed test function name>-<project name>/<dashed test function name>-<screenshot index>-(expected | actual | diff).png

Example: For the following file named content pages.spec.ts:

test('home page', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot();
}

The following test result images are generated when there is a failure for a project named iPad:

* ./test-results/content-pages-home-page-iPad/home-page-1-expected.png
* ./test-results/content-pages-home-page-iPad/home-page-1-actual.png
* ./test-results/content-pages-home-page-iPad/home-page-1-diff.png

Approved/golden images are stored in a directory named <spec file name>-snapshots using the following naming convention

./<path to spec file>-snapshots/<dashed test name>-<screenshot index>-<project name>-<platform identifier>.png

Here's an example assuming using the spec from above and assuming that it's located at ./e2e/content-pages.spec.ts and that the tests are run on macOS:

./e2e/content-pages.spec.ts-snapshots/home-page-1-iPad-darwin.png

Problem

The naming convention mismatch creates unnecessary cognitive overhead when copying an image from the ./test-results directory to the ./e2e/content-pages.spec.ts-snapshots directory. Users have to translate the naming convention used in the ./test-results directory to the naming convention that's used in the spec directory.

Proposed Solution

Change the naming convention in the ./test-results directory to match the naming convention in the project directory. So instead of the actual, diff, and expected files being named (using the examples above):

* ./test-results/content-pages-home-page-iPad/home-page-1-actual.png
* ./test-results/content-pages-home-page-iPad/home-page-1-expected.png
* ./test-results/content-pages-home-page-iPad/home-page-1-diff.png

The would be named:

* ./test-results/e2e/content-pages.spec.ts-snapshots/home-page-1-iPad-darwin-actual.png
* ./test-results/e2e/content-pages.spec.ts-snapshots/home-page-1-iPad-darwin-expected.png
* ./test-results/e2e/content-pages.spec.ts-snapshots/home-page-1-iPad-darwin-diff.png

Then all that would be required to "approve" the results of a failing screenshot test is running the following command:

cp ./test-results/e2e/content-pages.spec.ts-snapshots/home-page-1-iPad-darwin-actual.png ./e2e/content-pages.spec.ts-snapshots/home-page-1-iPad-darwin.png

Other Benefits

With approval being as simple as copying the file to the correct location, and all of the required information for doing so being embedded in the file path, then it would be much easier to develop a button in the test results UI for performing the copy without having to copy the file manually.

Related GitHub Issues

Related to #17801, #14883, #16970, and some of the discussion in #8161 (specifically this one).

[yury-s]

You can change location of the expected snapshots via snapshotPathTemplate, so something like this will be more similar to the actual snapshot path:

snapshotPathTemplate: '{testDir}/__screenshots__/{testFileDir}-{testFileName}-{testName}{-projectName}/{arg}-expected.{ext}',

will that work for you?

[mscottford]

I can experiment with that option and see how well it works. I do think that aligning the default file names would create a better out of the box developer experience. And it would also make it a little bit easier to create a button in the UI to handle approving the image.

[mscottford]

I looked at the options for snapshotPathTemplate and it is not possible to exactly replicate the naming convention used in the test-results directory. The closest I can get is by using

snapshotPathTemplate: '{testDir}/__screenshots__/{testFilePath}-{testName}{-projectName}/{arg}-expected{ext}',

which yields:

e2e/__screenshots__/content-pages.spec.ts-about-page-Desktop-Chrome/about-page-1-expected.png

There isn't a supported token that will yield just content-pages, which I'm guessing is the testFilePath without any extension.

Using this configuration value is better, but still not ideal.