playwright
playwright copied to clipboard
microsoft
[Feature] Playwright integration with Azure DevOps Test Plans
Is it possible to provide support for updating test results in Azure DevOps Test Plans by using Playwright directly ? I have come across this feature that is provided in Webdriver.IO. The test case execution status for Automated test cases are currently having to be updated manually as either passed or failed. And also the Automation status drop down we are not able to update it as Automated.
https://webdriver.io/docs/gmangiapelo-wdio-azure-devops-service https://github.com/gianlucamangiapelo/wdio-azure-devops-service/tree/main/.github/workflows
Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.
Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.
@dgozman It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?
It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?
Well, this is actually an integration between the test runner and ADO. @playwright/test for JS/TS includes a test runner, and this reporter integrates with ADO.
For python we recommend pytest, so you will need pytest+ADO integration, perhaps something like this?
Similarly, for Java you would need something that brings together JUnit+ADO, and for C# something that supports MSTest or NUnit with ADO.
It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?
Well, this is actually an integration between the test runner and ADO.
@playwright/testfor JS/TS includes a test runner, and this reporter integrates with ADO.For python we recommend
pytest, so you will need pytest+ADO integration, perhaps something like this?Similarly, for Java you would need something that brings together JUnit+ADO, and for C# something that supports MSTest or NUnit with ADO.
@dgozman I get it now. Playwright Test is only Js/TS Runner. I understand. So we need similar solutions for Pytest, NUnit and Junit5 as well now. I would appreciate if this can be looked at playwright team or some community members who can ? I see Azure DevOps has node api, python api, C# api and go api which can be used to build this integration. But i am not that skilled to do this. The pip library link you sent in previous reply will not solve it fully.
I get this in the log azure: Test [31035] Smoke test - Activity list @response - passed azure: Start publishing: [31035] Smoke test - Activity list @response SUITE PASS :) azure: Start upload attachments for test case [31035] azure: Result published: [31035] Smoke test - Activity list @response azure: Run 1000284 - Completed
but i see nothing in test case
I get this in the log
azure: Test [31035] Smoke test - Activity list @response - passed
azure: Start publishing: [31035] Smoke test - Activity list @response
SUITE PASS :)
azure: Start upload attachments for test case [31035]
azure: Result published: [31035] Smoke test - Activity list @response
azure: Run 1000284 - Completed
but i see nothing in test case
Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!
@Dgozman Any official update on integration playwright typescript/javascript with the ADO test plan ?? Here is a third-party reporter it just take a single plan id ( suite id ) at a time , in every sprunt the suite id changes , ( much appreciate the effort from the developer n thankyou! )
Is there any progression with playwright automated test results connect with azure devops manual test cases?
Yeah I'd also
Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!
Is there any place to track this, or any idea on when this integration might happen?
I get this in the log azure: Test [31035] Smoke test - Activity list @response - passed azure: Start publishing: [31035] Smoke test - Activity list @response SUITE PASS :) azure: Start upload attachments for test case [31035] azure: Result published: [31035] Smoke test - Activity list @response azure: Run 1000284 - Completed but i see nothing in test case
Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!
Do we have update on this one? I got the same result. My test case didn't update
Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.
@dgozman It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?
could you share some screenshots on how you wer able to work it?
Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.
Please note if you are committing your repo to GitHub make sure use secrets
Instead of passing token directly use secrets.
INSTEAD OF DOING:
token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',
DO:
token: process.env.ADO_TOKEN
to run it locally
export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds
FINALLY in GitHub Actions you can do like this
pass in
- name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test
Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.
Please note if you are committing your repo to GitHub make sure use secrets
Instead of passing token directly use secrets.
INSTEAD OF DOING:
token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',
DO:
token: process.env.ADO_TOKEN
to run it locally
export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds
FINALLY in GitHub Actions you can do like this
pass in
- name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test
I'm actually using Azure DevOps, I'm still struggling to update the result of a specific test case in my test plan in Azure Test Plan,
Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.
Please note if you are committing your repo to GitHub make sure use secrets
Instead of passing token directly use secrets.
INSTEAD OF DOING:
token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',
DO:
token: process.env.ADO_TOKEN
to run it locally
export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds
FINALLY in GitHub Actions you can do like this
pass in
- name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test
I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.
Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only. Please note if you are committing your repo to GitHub make sure use secrets Instead of passing token directly use secrets. INSTEAD OF DOING: token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds', DO: token: process.env.ADO_TOKEN to run it locally export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds FINALLY in GitHub Actions you can do like this pass in
- name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test
I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.
Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman
I am using a tool called SpecSync currently due to few problems I faced.
Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only. Please note if you are committing your repo to GitHub make sure use secrets Instead of passing token directly use secrets. INSTEAD OF DOING: token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds', DO: token: process.env.ADO_TOKEN to run it locally export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds FINALLY in GitHub Actions you can do like this pass in
- name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test
I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.
Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman
I am using a tool called SpecSync currently due to few problems I faced.
Yeah, this is what I am talking about, I want to update the outcome (result) of each test cases in my test plan in Azure Test Plan. Do we have already a solution for this? Does Playwright will support this thing soon? Any alternative that we can use?
@rowel @PavanMudigondaTR
Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only. Please note if you are committing your repo to GitHub make sure use secrets Instead of passing token directly use secrets. INSTEAD OF DOING: token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds', DO: token: process.env.ADO_TOKEN to run it locally export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds FINALLY in GitHub Actions you can do like this pass in
- name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test
I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.
Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman I am using a tool called SpecSync currently due to few problems I faced.
Yeah, this is what I am talking about, I want to update the outcome (result) of each test cases in my test plan in Azure Test Plan. Do we have already a solution for this? Does Playwright will support this thing soon? Any alternative that we can use?
Hello guys, Were you able to update the results to the test plans in Azure now ? If so, kindly share few inputs on how did you achieved it when you get sometime. Thanks!
getting this error while results got published,
this is my test case automation status in TFS
here is my log: azure: Uploading attachments for test: [31036] - Smoke test - Reports page @response @smok azure: Uploaded attachment azure: Left to publish: azure: Test results published for 14 test(s azure: Run 1000348 - Complete
where set the automation status and automation complexity? are these own fields no?
@daniavander ,
Please find these screenshots hope this helps
TFS Automation Options available
MyPlaywright Config
Test Cases -- see the error down below
Same Test Case ID i used
See these details too
After analyzing all these still something is missing or not sure where exactly it's going wrong if you get any idea great help.
@daniavander meanwhile please check this one provided by @alexneo2003
https://docs.specsolutions.eu/specsync/contact/troubleshooting#test-result-publishing-fails-with-mismatch-in-automation-status-of-test-case-and-test-run
so still trying to figure it out why this error is appearing. Please try to help if any ideas or hint that can resolve this issue Thanks.
@ravindrakumar6 sorry, i do not know what is the problem.
In ma devops i have only two status for automation status
in the config i do not add the suite id just the planid, perhaps
@aslushnikov are there any future time, when would you like to take this integration to the official release?
I tried asking questions of the playwright team in every livestream I could during Microsoft Build but it sounds like the Playwright team aren't familiar with Azure Test Plans. They kept referring to running tests in Azure Devops CI/Pipelines, but it doesn't sound like it will work with Azure Test Plans.
Would be good if you as a Microsoft team could support the Azure Test Plans feature.
I tried asking questions of the playwright team in every livestream I could during Microsoft Build but it sounds like the Playwright team aren't familiar with Azure Test Plans. They kept referring to running tests in Azure Devops CI/Pipelines, but it doesn't sound like it will work with Azure Test Plans.
Would be good if you as a Microsoft team could support the Azure Test Plans feature.
There are two options currently. I recommend you try out in below sequence.
This would be incredibly useful. I'm using the third-party reporter other people mentioned, at the moment https://github.com/alexneo2003/playwright-azure-reporter but a "built-in" integration would be awesome. Also integrated in the VSCode pluging, hopefully.
