The final report contains "Skipped" steps for tests, which were retested

[ilya-corp]

Hello! I have two passes of my tests: first - multithread (results into multi.json) and second - singlethread with retesting of failed scenarios (retest.json). During my second pass I generate SkipException for tests, which were passed at first pass. After finishing of both passes, I perform report generation:

                            E2E-Cucumber                             false                             target                             target                             *.json                             true                             true                             true                             true

Finally my report contains identical number of executed steps in the columns Skipped and Passed (see screenshot).

My expectation: Report must display execution time of every scenario and number of performed steps based on the following:

1. for successful and failed tests use the most recent data
2. for successful+skipped use data of successful
3. for failed+skipped use data of failed

So, finally: | Pass 1 | Pass 2 | Result | Use data of | | Success | Skipped | Success | Pass 1 | | Failed | Success | Success | Pass 2 | | Skipped | Success | Success | Pass 2 | | Failed | Failed | Failed | Pass 2 |

I don't need to see number of executed steps, performed at previous attempt, in the tests, which was retested. I need to see only number of retest steps.

Version of the plugin maven.cucumber.reporting: 5.7.0

[ilya-corp]

[damianszczepanik]

Attach JSON file. Are you talking about scenario or step which are executed for @Given ?

[ilya-corp]

I mentioned steps (Given, And, Then etc). As you can see, for main part of features contains identical number of performed steps. For example, for the first line of the screenshot you can see 18 passed steps and 18 skipped steps. It's because all 18 steps were successfully performed during first pass AND during second pass all steps of this scenario were skipped. So I suggest to leave only 18 passed steps and 0 skipped.

The JSON file is huge and contains some business data, do you really need it?

[damianszczepanik]

Have you tried this https://github.com/damianszczepanik/cucumber-reporting/blob/master/src/main/java/net/masterthought/cucumber/reducers/ReducingMethod.java#L64 ?

[ilya-corp]

I use both options

 <mergeFeaturesById>true</mergeFeaturesById>
 <mergeFeaturesWithRetest>true</mergeFeaturesWithRetest>

Is it a error to use them both?

[ilya-corp]

I have tried, but without positive result. Please see my new screenshot .

[ilya-corp]

 <configuration>
                            <projectName>E2E-Cucumber</projectName>
                            <skip>false</skip>
                            <outputDirectory>target</outputDirectory>
                            <inputDirectory>target</inputDirectory>
                            <jsonFiles>*.json</jsonFiles>
                            <setSkippedAsNotFailing>true</setSkippedAsNotFailing>
                            <skipEmptyJSONFiles>true</skipEmptyJSONFiles>
                            <mergeFeaturesWithRetest>true</mergeFeaturesWithRetest>
                        </configuration>

[damianszczepanik]

I use both options

 <mergeFeaturesById>true</mergeFeaturesById>
 <mergeFeaturesWithRetest>true</mergeFeaturesWithRetest>

Is it a error to use them both?

Honestly I have no idea how to answer for that question as most of those features were developed by guys who need them. I don't :) So my recommendation is to experiment and check...