microsoft
[Feature]: Label screenshots made in playwright with test file name
🚀 Feature Request
Append the sanitized test name to the automatic screenshot mechanism of Playwright.
Why?
Currently the names are very generic and cause conflicts in CI/CD Systems like Azure Devops when we have multiple jobs failing with the same generic name:
It's also not very clear which number responds to which testcase when we are able to upload:
Example
Fix!
Added the sanitized file name, a good example of how this looks like are the fixed unit tests :
This will allow us to find the artifact easier and not conflict in CI/CD systems like in Azure Devops
Motivation
This is purely a CI/CD Improvement for Playwright integration for us, we have a massively parallel system running hundreds of tests across multiple shards
@mastrzyz Thank you for filing the issue!
Don't you have the same problem with traces, all named trace.zip? It seems like the better solution is to update your reporter to include the test name in any attachments it generates, since these are presented as a plain list, not grouped by test.
Sorry @dgozman , I'm not using any custom reporter and relying on the existing Playwright reporters which are outputting trace.zip and screenshot1.png.
@mastrzyz In this case, why do you only focus on the screenshots? I'd think that trace.zip would be equally important. Do I miss something?
If I understand correctly, you are using junit reporter that is uploaded to Azure DevOps through this task, don't you? Perhaps we can improve that part instead? Why does it not show the full attachment path? Reading this post, I would think the full attachment path that already includes test path could be shown. Not to mentioning that it's a part of the specific test output.
You know to tell you the truth, why don't I see this failure for trace.zip!
Could it be that trace.zip is only one? we have retries enabled so I wonder if we take screenshots in first attempt and then seccond also takes and collides
@mastrzyz Perhaps you should set both trace and screenshot to 'on-first-retry'?
: { trace: "on-first-retry", video: "on-first-retry" }), yeah we have that already
: { trace: "on-first-retry", video: "on-first-retry" }), yeah we have that already
Sorry, I do not quite understand. The snippet does not show the screenshot setting. I think that if you set it to on-first-retry similarly to trace/video, your warnings will disappear. Could you please check that?
@dgozman , I don't think that is possible ->
but I Get what you mean, this would solve it
@mastrzyz Indeed, my mistake. I think we can introduce screenshot: 'on-first-failure' mode that would limit the total number of screenshots per test to one, and most likely fix your issue.
