cucumber-jvm

cucumber-jvm copied to clipboard

🤔 What's the problem you're trying to solve?

I'm a team lead on a team currently implementing cucumber, and one of our usecases is being able to run individual features and scenarios. This is well supported, but to ensure this will work for any items, the features should be ran in a random (probably seeded) order, to highlight scenario dependencies. This is available in most versions of cucumber across languages, as well as in the cucumber core for java, but there is no way to access it in cucumber engine.

✨ What's your proposed solution?

Based on the existing property names and namespace for cucumber engine, and this feature in other environments that have it, introduce property cucumber.execution.order into cucumber engine. Cucumber engine already forwards all properties starting with "cucumber." from all sources so getting it should be smooth. From there its value should be mapped onto the correct PickleOrder value. Default should preserve existing behavior (lexical). Other options are random, random(seeded) and reverse lexical. I need random at least, but random(seeded) would be better.

⛏ Have you considered any alternatives or workarounds?

It's not usually recommended to hack into 3rd party libraries from your testcode, which leaves only hightened manual vigilance as an option.

📚 Any additional context?

Would be happy to try implementing it myself if your team will have the PR.

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

I would be happy to accept a pull request for this!

You can add a step to the DiscoverySelectorResolver that randomizes the features.

https://github.com/cucumber/cucumber-jvm/blob/edf09f1fbabf0a9bb2cb0d3abab59eb52875585f/cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/DiscoverySelectorResolver.java#L46-L51

To randomize the pickles, be aware that the JUnit Platform represents tests in a tree. For implementation hints, please have a look at JUnit Jupiter:

https://github.com/junit-team/junit5/blob/d4adab8e08e9c73ecbfdef21638cfe3022c61fe5/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/discovery/DiscoverySelectorResolver.java#L41C1-L42

And specifically the comments at:

https://github.com/junit-team/junit5/blob/d4adab8e08e9c73ecbfdef21638cfe3022c61fe5/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/discovery/AbstractOrderingVisitor.java#L99-L101

Note that currently we don't use the EngineDiscoveryRequestResolver but will after https://github.com/cucumber/cucumber-jvm/pull/2835 is done. This may require us to resolve some merge conflicts, but I don't think they'll be significant.

And pulling #2835 back into memory I see that I already had to implement some ordering and stalled there. So you may want to consider building on that PR instead. But I would accept either one of these solutions.

• https://github.com/cucumber/cucumber-jvm/pull/2835/files#diff-af3f8724cf1982da8deef60f2f456283064f0cd3f76ecc596773db9c30bd6900
• https://github.com/cucumber/cucumber-jvm/pull/2835/files#diff-11bbbf877c5770e5d2398450c994d73ae34264e40e3ef0ec19474e60a1cb2090