[BUG] Incorrect path for expected image on attachments

[edumserrano]

Hi @cenfun,

I think I've hit a bug regarding the display of the Expected image under the attachments section of a test. This is what I'm getting:

What I have noticed

1. Both the Diff and Actual are displaying the image properly.
2. Both the Diff and Actual point to the images that are inside playwright's outputDir which for me is ./tests/test-results:
   • Diff image is being fetched from http://localhost:8090/example-screenshot-chromium/screenshot-1-diff.png.
   • Actual image is being fetched from http://localhost:8090/example-screenshot-chromium/screenshot-1-actual.png.
3. The folder example-screenshot-chromium exists at ./tests/test-results and contains the 3 images, the Diff, Actual and Expected as shown by:

4. Unlike the Diff and Actual images, the Expected image is being fetched from http://localhost:8090/example.spec.ts-snapshots/screenshot-1-chromium-win32.png. I would expect it would try to fetch from http://localhost:8090/example-screenshot-chromium/screenshot-1-expected.png. It seems it's trying to get the expected from the actual folder where the expected snapshots are saved, which for this test would be at ./tests/example.spec.ts-snapshots/screenshot-1-chromium-win32.png. However, with my playwright configuration, this folder is not being served and is inaccessible from the html report.

How to reproduce

1. Clone the tag temp-1 from my playwright-adventures repo.
2. With your favourite editor go to ./demos/code-coverage-with-monocart-reporter/src/app/app.component.css and change the line 2 from color: green; to color: red;.
3. Use your favourite shell to go to ./demos/code-coverage-with-monocart-reporter.
4. Run npm i to install all required npm packages.
5. Run npm run test to run the playwright tests. 3 tests should fail, 6 should pass.
6. Run npx monocart show-report tests/test-results/index.html and inspect the Expected image of any of the failed tests.

Further notes

1. This is happening on version 1.7.13 of the monocart-reporter package and using @playwright/test version 1.40.1.
2. Perhaps the problem is with my playwright.config.ts and how I configured the output directories. I still think it should work and that the expected image is not being fetched from the playwright's output dir.

[cenfun]

Seems it is same with this issue: https://github.com/cenfun/monocart-reporter/issues/49 Basically because the path is too long on Windows, see maximum-file-path-limitation What is your OS? 1, First, I can NOT reproduce it locally by default. 2, But after rename the folder name test-results to test-results-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-long-, then reopen the report and the images failed to display.

Can you try to shorten the path?

[edumserrano]

I'm sorry but that can't really be it can it?

1. Both the Diff and Actual images show properly. Only the Expected is not showing. All 3 images are in the same folder and therefore have the same folder path restriction, if it works for 2 of them it should work for the third right?
2. If instead of http://localhost:8090/example.spec.ts-snapshots/screenshot-1-chromium-win32.png, which is the failing path that is completely wrong since it's not even pointing to the test results folder, I use http://localhost:8090/example-screenshot-chromium/screenshot-1-expected.png then the image shows up on the browser.
3. The full path to the expected file on my machine is C:\dev\repos\github-personal\playwright-adventures\demos\code-coverage-with-monocart-reporter\tests\test-results\example-screenshot-webkit\screenshot-1-expected.png which is 165 characters long.
4. Here's a zip of my test-results folder produced from the repo above on tag temp-1 in case it helps debug: test-results.zip
5. Not sure it matters but I'm using Windows 11 Version 22H2 (OS Build 19045.3693).

[cenfun]

sorry, it should not be same issue. I can reproduce it with test-results.zip your provided. there is a image not in the test-results folder

.../tests/test-results/example-screenshot-chromium/screenshot-1-diff.png
.../tests/test-results/example-screenshot-chromium/screenshot-1-actual.png
.../tests/example.spec.ts-snapshots/screenshot-1-chromium-win32.png

let me check why the path is confusing

[cenfun]

When test with @playwright/test version 1.36.2, the path are correct as expected

.../tests/test-results/example-screenshot-chromium\screenshot-1-expected.png
../tests/test-results/example-screenshot-chromium\screenshot-1-actual.png
../tests/test-results/example-screenshot-chromium\screenshot-1-diff.png

But if updated @playwright/test version to 1.39 or 1.40, the path of expected image has changed

.../tests/example.spec.ts-snapshots/screenshot-1-chromium-win32.png
.../tests/test-results/example-screenshot-chromium/screenshot-1-actual.png
.../tests/test-results/example-screenshot-chromium/screenshot-1-diff.png

I found the change why expected image path is different: fix: use snapshotPath instead of expectedPath for image diffs Let me think about how to solve it ...

[edumserrano]

I've tested with the recently released v2.0.0 of monocart-reporter and now the image is showing up in the attachments but the link to it, the link that shows under the images is still pointing to the incorrect path as shown in this image:

In case it matters, you can check this behavior with the v2 of the package by cloning the temp-2 tag from the playwright-adventures repo and running npm run test from the `demos/code-coverage-with-monocart-reporter/ folder. Note that in this tag there are 3 tests failing and it's expected. After running the tests you can check any of the failed tests for the incorrect link to the expected snapshot.

Also, here's a zip of the test-results test-results.zip

[cenfun]

good catch, please try version 2.0.1 currently, just replace the expected path with hard-code way in UI. It should be a bug of Playwright, the expected image should be in the same dir as diff/actual, consider submmitting an issue to playwright.

[edumserrano]

I've updated to version 2.0.1 and now the link for the expected image is also working. 👏

I can open an issue on Playwright but I'm not sure what to say. Could you provide me with a bit more information on why you're saying this is a Playwright bug?