cucumber-jvm

cucumber-jvm copied to clipboard

🤔 What's the problem you're trying to solve?

Given a Scenario Outline described as "Orders with a total value > <value> receive a <discount>% discount"
And an example where <value> is 1000 and <discount> is 2
And the configured JUNIT_PLATFORM_NAMING_STRATEGY_PROPERTY_NAME is "long"
When I run the scenario outline using the junit-platform-engine
Then the reported example name should be "Orders with a total value > 1000 receive a 2% discount"

✨ What's your proposed solution?

A brief look at the code shows NamingStrategy.LONG already traverses up the parent nodes and potentially could be extended to replace the variables. This already works in cucumbers own CLI output, e.g. the Pretty Formatter, but I couldn't find the code where the actual substitution takes place with a short search.

It might be good to look at a somewhat exhaustive example:

Feature: Belly

  Scenario Outline: <value> cukes
    Given I have 42 cukes in my belly
    When I wait 1 hour
    Then my belly should growl

    Examples: A
      | value |
      | 1     |
      | 2     |

    Examples: B
      | value |
      | 3     |
      | 4     |

Is rendered as:

Belly
Belly - <value> cukes
Belly - <value> cukes - A
Belly - <value> cukes - A - Example #1
Belly - <value> cukes - A - Example #2
Belly - <value> cukes - B
Belly - <value> cukes - B - Example #1
Belly - <value> cukes - B - Example #2

Rendering this as:

Belly
Belly - <value> cukes
Belly - <value> cukes - A
Belly - 1 cukes - A - Example #1
Belly - 2 cukes - A - Example #2
Belly - <value> cukes - B
Belly - 3 cukes - B - Example #1
Belly - 4 cukes - B - Example #2

May lead people to wonder where the other examples are for "1 cukes". Especially when non-numerial values are used. We can't quite omit the - A - Example #1 part because not all scenario outlines have to use parameters.

And rendering as:

Belly
Belly - <value> cukes
Belly - <value> cukes - A
Belly - <value> cukes - A - Example #1 - 1 cukes
Belly - <value> cukes - A - Example #2 - 2 cukes
Belly - <value> cukes - B
Belly - <value> cukes - B - Example #1 - 3 cukes 
Belly - <value> cukes - B - Example #2 - 4 cukes

Would significantly increase the length and adds duplication.

So perhaps you could you clarify in which context you're viewing the results? And what problems you are running into? The naming strategy is a workaround to deal with limitations of Surefire and Gradle when it comes to reporting. So it is important to consider changes in that context.

This already works in cucumbers own CLI output, e.g. the Pretty Formatter, but I couldn't find the code where the actual substitution takes place with a short search.

There is a subtle difference between what Pretty formatter and JUnit 5 report on. And there is an impedance mismatch between the two methods of reporting.

Before execution, Cucumber compiles a feature file with scenarios and scenario outline examples into a flat list of pickles. Pickles are "test cases" derived from a feature file. So the full hierarchy is:

Feature (Belly)
\- Scenario Outline (<value> cukes)
 \- Examples (A)
  \- Example (Example #1)
   \- Pickle (1 cukes)

The pretty plugin reports on the execution of the pickle Pickle (1 cukes) as they're executed while JUnit 5 reports on the entire hierarchy leading up to the pickle Feature (Belly) - Scenario Outline (<value> cukes) - Examples (A) - Example (Example #1).

Thanks for adding the more exhaustive example!

So perhaps you could you clarify in which context you're viewing the results? And what problems you are running into?

My main use case is running the test from IntelliJ, which renders the JUnit results as a tree in the UI exactly like in the example you gave. My - really minor - annoyance is when one of the examples fails, it's a bit harder to understand which one that that is since the failing one is just presented as Example #n and one needs to select the example in the tree and look at the console output to see it's context.

Assuming this would make sense to you:

Belly
 \- <value> cukes
  \- - A
    \- Example #1 - 1 cukes

you could change namingStrategy.name(node) to namingStrategy.name(node, pickle) by adding an extra method and introduce a new naming strategy.

https://github.com/cucumber/cucumber-jvm/blob/250617a557ff5715472708670bf4c075e6ebe345/cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/FeatureResolver.java#L121-L131

Hello, I have a same change request. It is currently not very intuitive which test has failed. I have 500+ tests in application, so it takes time to find corresponding Example that failed if the table is large.

I have implemented possible solution that works fine for my tests. I’m sure that it can be done more efficiently but it may serve as proof of concept. The Pickle Object basically contains resolved string for each Pickle witch I then use to replace Example #x string.

Here is my branch: https://github.com/bjelac/cucumber-jvm/tree/feature/2598_resolve_example_variables_in_scenario_outline_name_in_junit-platform-engine

Before change - strategy long:

After change - strategy long:

Before change - strategy short:

After change - strategy short:

So please check my branch and give me your thoughts. Am open for any suggestion/change that makes more sense.

It seems like this may be a suitable point for an extension.

Following JUnit 5 pattern for configuring parallism we could do something like cucumber.junit-platform.naming-strategy=custom and a matching cucumber.junit-platform.naming-strategy.config.custom.class=SomeStrategy.

The SomeStrategy class would implement and interface. The tricky part will be ensuring that this is a clean API that doesn't expose Cucumbers internals.

@bjelac @ChrisAtIonos would either of you be interested in creating a PR for this?

Please feel free to reopen this if there is an interest in providing a pull request.