[Feature] Playwright test-runner integration with TestRail

[mnataraj17]

Let us know what functionality you'd like to see in Playwright and what your use case is. Do you think others might benefit from this as well?

I would like to integrate playwright test results with TestRail. I can see some libraries for integrating jest with TestRail but If I use Jest (not Playwright test runner) I will be missing out on lot of features playwright is offering. Most of the companies try to integrate their automation test results with their test management tools. It would be great if playwright team works on this feature. Thanks!

[pavelfeldman]

We don't have immediate plans to implement it, and the implementation is unlikely to end up in our repo, but I'll leave it open to collect upvotes.

[unlikelyzero]

I did some poking around and found that TestRail does not have an official node sdk. The closest I found was here: https://github.com/dlenroc/node-testrail-api

I would think that the fastest way to a working prototype would be for someone to work off of the result here: https://github.com/dlenroc/node-testrail-api

@mnataraj17 which CI system are you using? Jenkins?

[mnataraj17]

@unlikelyzero Thanks for the info. we are using Github Actions

[unlikelyzero]

@mnataraj17 I'm amazed that there isn't an example of junit -> testrail for GHA.

I'd go with this implementation https://github.com/JSanchezIO/junit-testrail-reporter

1. Get your playwright tests to output https://github.com/JSanchezIO/junit-testrail-reporter
2. Then npx junit-testrail-reporter -p='test-results/*.xml'

Bonus points if you share your working example with the larger Playwright community!

[reach2jeyan]

@mnataraj17 did you get a chance to implement the above? I am trying the same scenario, where I have a bunch of tests in playwright with the junit report and like @unlikelyzero said, i tried implementing, but it somehow keeps throwing me the error Unexpected token '?'.

Not really sure where am I going wrong.

This is the command i m trying to

npx junit-testrail-reporter -p=test-results-o.xml -h=TESTRAIL_HOST -TESTRAIL_PASSWORD=TESTRAIL_PASSWORD -projectId=1 -u=username

the Documentation does not have an argument for the TESTRAIL_PASSWORD

[mnataraj17]

@reach2jeyan I have not tried it yet, working on some other priority items. Sorry for not being able to help you here right now.

[unlikelyzero]

@reach2jeyan Do you have a known good working j unit file for your organization? That would be where I would start first to make sure that you can actually push to test rail without playwright being involved

[reach2jeyan]

@unlikelyzero thanks for getting back. Yes I do have a junit file, and It works when I try to view the results of the tests in the jenkins so I consider it working. So I tried to use the junit test rail reporter library, but I am stuck with the issue that I posted with unrecognised token.

Looks like I have to write a custom hook, as that library is not working for me. But I had like to confirm as well if I am passing the right arguments to make it work?

[unlikelyzero]

@reach2jeyan hrmm. From their docs

The password configuration parameter can only be set via environment variable

[reach2jeyan]

@unlikelyzero

This is the command I used

./node_modules/.bin/junit-testrail-reporter -p=test-results-0.xml -h=TESTRAIL_HOSTD -projectId=1 -u='username'

Is this correct?

I get this error like SyntaxError: Unexpected token '?'

I am passing the password arguments by export TESTRAIL_PASSWORD=mypassword

but the error when I navigated to the code feels like this

default: process.env.TESTRAIL_RUN_NAME ?? 'Automated Test Run via junit-testrail-reporter', in https://github.com/JSanchezIO/junit-testrail-reporter/blob/f8b521c9efb8e48a8eb574ca7f7d441eb4a3f319/src/cli.ts#L53

so i am not really sure if I am passing the right args or is it about the library that is throwing the error. and hence I commented out the ?? and defaulted it to the hardcoded text, but then other errors came by as well and hence i stopped getting into, because i really did not know if I am going in the right direction, felt I could just write a direct hook using the node library.

but I want to hear your suggestions too.

[unlikelyzero]

@reach2jeyan I would maybe take this bug/question to https://github.com/JSanchezIO/junit-testrail-reporter to get that half of the implementation solved. I'd imagine that getting this fixed on that project would be far faster than trying to role something.

Also, there is a official jenkins plugin. If you're doing this from Jenkins, you could just use https://github.com/jenkinsci/testrail-plugin

So, get the junit result from https://playwright.dev/docs/test-reporters#junit-reporter

Then use Jenkins testrail plugin to push results to testrail

[reach2jeyan]

@unlikelyzero yup! I had created a support ticket in the respective repository. and wow! I did not know about this plugin, Yes! I am using Jenkins,I was quite skeptical to use Pangolin, this comes in pretty life and time saver as we already use the junit of playwright. thanks a ton!, i m gonna try this out tomorrow

[reach2jeyan]

@unlikelyzero well i hit a snag in the official plugin as well its not pretty much in usable state since long time due to this open null pointer issue haha https://github.com/jenkinsci/testrail-plugin/issues/64

[reach2jeyan]

@unlikelyzero

This is the command I used

./node_modules/.bin/junit-testrail-reporter -p=test-results-0.xml -h=TESTRAIL_HOSTD -projectId=1 -u='username'

Is this correct?

I get this error like SyntaxError: Unexpected token '?'

I am passing the password arguments by export TESTRAIL_PASSWORD=mypassword

but the error when I navigated to the code feels like this

default: process.env.TESTRAIL_RUN_NAME ?? 'Automated Test Run via junit-testrail-reporter', in https://github.com/JSanchezIO/junit-testrail-reporter/blob/f8b521c9efb8e48a8eb574ca7f7d441eb4a3f319/src/cli.ts#L53

so i am not really sure if I am passing the right args or is it about the library that is throwing the error. and hence I commented out the ?? and defaulted it to the hardcoded text, but then other errors came by as well and hence i stopped getting into, because i really did not know if I am going in the right direction, felt I could just write a direct hook using the node library.

but I want to hear your suggestions too.

Just wanted to update here that

This issue has been fixed with the version v1.1.1 https://github.com/JSanchezIO/junit-testrail-reporter/issues/1

[reach2jeyan]

@unlikelyzero Looks like implementing our own custom service in our framework that uses playwright tool is much easier than using of the Jenkinsfile , I have a working example that have implemented in the framework.

[mnataraj17]

@reach2jeyan it would be great if you could post details of your implementation. :)

[unlikelyzero]

@reach2jeyan that's great to hear! I would recommend that your team moves off of node12 ASAP. It hit's EOL this month

[mnataraj17]

@reach2jeyan Are you using https://github.com/JSanchezIO/junit-testrail-reporter in your framework or have written any custom code using Testrail APIs?

[ButeoButeo]

Will be great to have Playwright test-runner integration with TestRail for update test results. Hope this could be a reality today.

[ButeoButeo]

@unlikelyzero Looks like implementing our own custom service in our framework that uses playwright tool is much easier than using of the Jenkinsfile , I have a working example that have implemented in the framework.

@unlikelyzero, It would be great if you could post details of your implementation. I'm using GitHub Actions. Thanks You!

[unlikelyzero]

@ButeoButeo I did not implement anything myself. @reach2jeyan was able to get a successfull integration

[ButeoButeo]

@reach2jeyan did you use the xml report generated by Playwright test runner? Thanks !

[DimaKramskoy]

we have achieved the integration using custom reporter , written using the Playwright documentation . the integration is work in progress , so far works well . we use this npm to work with gurock api https://www.npmjs.com/package/testrail-api so far we're missing the ability of upload screenshots and the ability to add the videos . this is the next step.. as an alternative would be interesting to see if anyone was able to integrate using the result xml file , in one batch upload if this option exists.. Thank you!

[LiaMoreira]

Hi everyone! How is this integration going? I need to integrate Playwright with Test Rails because there's where the team manages tests, but I couldn't do it so far.

[cenfun]

Simply we can submit playwright test results to testrail using node-testrail-api

example here: https://github.com/cenfun/monocart-reporter-test/tree/main/integrations/testrail

[denisha]

I have developed a fundamental implementation for this by utilizing the TestRail API library. It has been tailored specifically to meet my company's requirements, so it may not include all advanced features, but it effectively addresses the core issue.

You can find the NPM package here: https://www.npmjs.com/package/playwright-testrail-reporter

Additionally, I have written a blog post about it: https://dev.to/denisha/playwright-to-testrail-reporter-for-test-automation-1go6

[tsakrikas]

Hi @denisha, could you please explain how you declare and use the required environment variables?

[denisha]

@tsakrikas you just need to create a .env file and declare the environment variables mentioned in the readme of my repo in that file. if you have further issues please log them directly on the repo

[hatufacci]

Hey, @denisha i didn't find /close_run call in your lib https://support.testrail.com/hc/en-us/articles/7077874763156-Runs#closerun it will be nice to have boolean variable for that, not closed runs marked with is_completed: false in testRail