[🐛 Bug]: @wdio/junit-reporter in WebDriverIO v8 with CucumberJS shows step counts instead of test counts in JUnit report

[bog777bog]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

WebdriverIO Version

8.20.4

Node.js Version

18.18.2

Mode

WDIO Testrunner

Which capabilities are you using?

{
browserName: 'chrome',
          'wdio:devtoolsOptions': {
              headless: false
          }

}

What happened?

The JUnit reporter in WebDriverIO with CucumberJS is producing reports that display the count of individual test steps instead of counting complete test scenarios.

What is your expected behavior?

Each test scenario (each "Given," "When," "Then" sequence) should be counted as a single test case in the JUnit report, providing a clear and accurate overview of the test coverage.

How to reproduce the bug.

STR:

1. Clone Offical boilerplate https://github.com/amiya-pattnaik/webdriverIO-with-cucumberBDD
2. npm install
3. Enable Juinit reporter in wdio.shared.conf.ts
4. Run tests npm run test-local
5. Go to reports folder
6. Open JUinit report
7. Observe generate JUnit report at test/reports/junit-results

actual state: The JUnit reporter in WebDriverIO with Cucumber is producing reports that display the count of individual test steps instead of counting complete test scenarios.

<testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:07" time="2.554" tests="3" failures="0" errors="0" skipped="0">

expected state: Each test scenario (each "Given," "When," "Then" sequence) should be counted as a single test case in the JUnit report, providing a clear and accurate overview of the test coverage.

<testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:07" time="2.554" tests="1" failures="0" errors="0" skipped="0">

Relevant log output

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:07" time="2.554" tests="3" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="As a user, I can log into the secure area"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/login.feature"/>
    </properties>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Given I am on the login page" time="1.956">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/url - {"url":"https://the-internet.herokuapp.com/login"}
RESULT: {"url":"https://the-internet.herokuapp.com/login"}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="When I login with tomsmith and SuperSecretPassword" time="0.556">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#username"}
RESULT: {"using":"css selector","value":"#username"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_3/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_3/value - {"text":"tomsmith"}
RESULT: {"text":"tomsmith"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#password"}
RESULT: {"using":"css selector","value":"#password"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_4/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_4/value - {"text":"SuperSecretPassword!"}
RESULT: {"text":"SuperSecretPassword!"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"button[type=\"submit\"]"}
RESULT: {"using":"css selector","value":"button[type=\"submit\"]"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_10/click - {}
RESULT: {}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Then I should see a flash message saying You logged into a secure area" time="0.038">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/elements - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: GET /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/01B1D3E994A64795A27A65935ECF9044_element_13/text - {}
RESULT: {}
]]></system-out>
    </testcase>
  </testsuite>
  <testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:09" time="0.591" tests="3" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="As a user, I can log into the secure area"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/login.feature"/>
    </properties>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Given I am on the login page" time="0.165">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/url - {"url":"https://the-internet.herokuapp.com/login"}
RESULT: {"url":"https://the-internet.herokuapp.com/login"}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="When I login with foobar and barfoo" time="0.393">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#username"}
RESULT: {"using":"css selector","value":"#username"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_17/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_17/value - {"text":"foobar"}
RESULT: {"text":"foobar"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#password"}
RESULT: {"using":"css selector","value":"#password"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_18/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_18/value - {"text":"barfoo"}
RESULT: {"text":"barfoo"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"button[type=\"submit\"]"}
RESULT: {"using":"css selector","value":"button[type=\"submit\"]"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_25/click - {}
RESULT: {}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Then I should see a flash message saying Your username is invalid" time="0.031">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/elements - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: GET /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/E7F58260FE635C38EEEAEB1F0B9FD73C_element_28/text - {}
RESULT: {}
COMMAND: GET /session/a021b7b72ba5c7c898ad991cca6a8d3e/se/log/types - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/se/log - {"type":"driver"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/se/log - {"type":"browser"}
RESULT: {"type":"driver"}
RESULT: {"type":"browser"}
COMMAND: DELETE /session/a021b7b72ba5c7c898ad991cca6a8d3e - {}
RESULT: {}
]]></system-out>
    </testcase>
  </testsuite>
  <testsuite name="The Internet Guinea Pig Website" timestamp="2023-11-01T21:55:07" time="3.15" tests="0" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="The Internet Guinea Pig Website"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/login.feature"/>
    </properties>
  </testsuite>
</testsuites>

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• [X] I have searched the existing issues

[christian-bromann]

Each test scenario (each "Given," "When," "Then" sequence) should be counted as a single test case in the JUnit report, providing a clear and accurate overview of the test coverage.

Why is this given a better and more clear overview of the test coverage? How do you consume the Junit report and why is the current structure not meeting your needs?

[bog777bog]

@christian-bromann We rely on Azure DevOps and the PublishTestResults@2 task in our CI/CD pipeline for accurate test reporting. Prior to migrating to WebDriverIO v8, the JUnit reports correctly treated each Cucumber scenario as a single test case, providing the right count of test cases.

However, post-migration, the JUnit reports now display individual test steps instead of test cases. This has disrupted our test management and reporting processes, making it difficult to assess test coverage, track issues, and make informed release decisions in Azure DevOps.

PublishTestResults@2 https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-test-results-v2?view=azure-pipelines&tabs=trx%2Ctrxattachments%2Cyaml

For instance, prior to the migration, it used to show 400 test scenarios, but now it presents thousands of individual steps. Which is really challenging to track.

[christian-bromann]

@bog777bog do you have a WebdriverIO JUnit test report prior and after migration?

[bog777bog]

@christian-bromann there's a noticeable difference of JUnit report before and after migrating from WebdriverIO v7 to v8

Before:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="Google" timestamp="2023-11-01T23:11:46" time="1.531" tests="1" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="featureName" value="Google"/>
      <property name="capabilities" value="chromeheadless.118_0_5993_117.darwin"/>
      <property name="featureFile" value="./test/features/bau/desktop/tmp/google_1.feature"/>
    </properties>
    <testcase classname="CucumberJUnitReport-chromeheadless.118_0_5993_117.darwin.Google" name="User visits Google page" time="1.47">
      <system-out><![CDATA[
✅ Given I visit Google page on us✅ Then I expect to be on the Google page
]]></system-out>
    </testcase>
  </testsuite>
</testsuites>

After:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="User visits Google page" timestamp="2023-11-01T23:07:07" time="0.874" tests="2" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="User visits Google page"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/bau/desktop/tmp/google_1.feature"/>
    </properties>
    <testcase classname="chrome.118_0_5993_117.mac.google_1.feature:3:1:_User_visits_Google_page" name="Given I visit Google page on us" time="0.849">
      <system-out><![CDATA[
COMMAND: POST /session/2b824db7f9b4c6993a585fee742e0259/url - {"url":"https://www.google.com/"}
RESULT: {"url":"https://www.google.com/"}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.google_1.feature:3:1:_User_visits_Google_page" name="Then I expect to be on the Google page" time="0.013">
      <system-out><![CDATA[
COMMAND: GET /session/2b824db7f9b4c6993a585fee742e0259/url - {}
RESULT: {}
COMMAND: DELETE /session/2b824db7f9b4c6993a585fee742e0259 - {"deleteSessionOpts":{"shutdownDriver":false}}
RESULT: {"deleteSessionOpts":{"shutdownDriver":false}}
COMMAND: DELETE /session/d79df7fc5fee413c81de752c5584fb6d - {}
RESULT: {}
]]></system-out>
    </testcase>
  </testsuite>
  <testsuite name="Google" timestamp="2023-11-01T23:07:07" time="0.911" tests="0" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="Google"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/bau/desktop/tmp/google_1.feature"/>
    </properties>
  </testsuite>
</testsuites>

[christian-bromann]

Thanks for providing these logs, it draws a clear picture. I assume someone has added more detail to the JUnit reporter because they had a need to have more details per step execution. I see that we have controls for this behavior in the Allure reporter.

My suggestion is to allow to define this behavior as reporter configuration and let people choose if they like to define every step as a test case or ever scenario.

Any contributions are appreciated!

[MikeBaxter-Canopy]

@bog777bog @christian-bromann We have the same issue since upgrading to WDIO v8.

Is this being actively worked on? If so have you managed to find any workaround to get the JUnit reports back to the way they output previously?

[christian-bromann]

Is this being actively worked on?

No one is assigned to this issue, so if you like to take a stab, feel free.

[MikeBaxter-Canopy]

I've been trying to get it working locally for the last few days without any success so I don't have the technical know-how to fix it sadly. That's why I was hoping someone else would have figured out a way to fix it.

When I've looked through the @wdio/junit-reporter docs https://webdriver.io/docs/junit-reporter/#configuration it doesn't look like there's a lot of configuration that could be altered.

[christian-bromann]

@MikeBaxter-Canopy my suggestion above was to add an option to change the behavior. I honestly don't know why the current behavior is not desired since the report contains more information than before. Can you elaborate?

[bog777bog]

@christian-bromann @MikeBaxter-Canopy yes, but it shows things differently like tests are not equal to steps. Other clients utilizing this junit reporter heavily depend on this attribute for their workflows.

[christian-bromann]

but it shows things differently like tests are not equal to steps.

Can you explain on an actual example? The report above seems ok to me 🤔

[bog777bog]

@christian-bromann I've already provided the comparison of these reports above. Where you may observe that WDIO v7 shows tests="1", meanwhile WDIO v8 shows tests="2". As you can see, it returns the amount of steps instead of tests.

[christian-bromann]

As you can see, it returns the amount of steps instead of tests.

Yes, can you clarify why this is not the desired behavior? A step can be seen as a test since there can be more than one command being executed.

[BorisOsipov]

Yes, can you clarify why this is not the desired behavior? A step can be seen as a test since there can be more than one command being executed.

It is simple. The official cucumber-js junit formatter counts scenarious as junit testcases and steps are nicely printed in <system-out> section for corresponding testcase.

[christian-bromann]

Thanks for clarifying @BorisOsipov! I think nothing speaks to offering a compliant version with the native Cucumber JUnit reporter. As I suggested before, the best way would be to add a reporter configuration and change the behavior based on on that. Any contributions to this are much appreciated!