cucumber-jvm

cucumber-jvm copied to clipboard

Many users discover Cucumber Reports by reading the banner printed at the end of the run:

┌───────────────────────────────────────────────────────────────────────────────────┐
│ Share your Cucumber Report with your team at https://reports.cucumber.io          │
│ Activate publishing with one of the following:                                    │
│                                                                                   │
│ src/test/resources/cucumber.properties:          cucumber.publish.enabled=true    │
│ src/test/resources/junit-platform.properties:    cucumber.publish.enabled=true    │
│ Environment variable:                            CUCUMBER_PUBLISH_ENABLED=true    │
│ JUnit:                                           @CucumberOptions(publish = true) │
│                                                                                   │
│ More information at https://cucumber.io/docs/cucumber/environment-variables/      │
│                                                                                   │
│ Disable this message with one of the following:                                   │
│                                                                                   │
│ src/test/resources/cucumber.properties:          cucumber.publish.quiet=true      │
│ src/test/resources/junit-platform.properties:    cucumber.publish.quiet=true      │
└───────────────────────────────────────────────────────────────────────────────────┘

The instructions suggest that one way to enable publishing is to add cucumber.publish.enabled=true to src/test/resources/junit-platform.properties. However, it isn't obvious that defining the environment variable will have no effect with JUnit 5.

This might come as a surprise to users who are upgrading from v6 to v7 and trying out the new JUnit 5 runner.

I propose that we modify the banner to make it more explicit what activation mechanism work with what runners:

┌───────────────────────────────────────────────────────────────────────────────────┐
│ Share your Cucumber Report with your team at https://reports.cucumber.io          │
│ Activate publishing with one of the following:                                    │
│                                                                                   │
│ JUnit 5:                                                                          │
│ src/test/resources/junit-platform.properties:    cucumber.publish.enabled=true    │
│ JUnit 4 / TestNG:                                                                 │
│ src/test/resources/cucumber.properties:          cucumber.publish.enabled=true    │
│ Environment variable:                            CUCUMBER_PUBLISH_ENABLED=true    │
│                                                                                   │
│ More information at https://cucumber.io/docs/cucumber/environment-variables/      │
│                                                                                   │
│ Disable this message with one of the following:                                   │
│                                                                                   │
│ src/test/resources/junit-platform.properties:    cucumber.publish.quiet=true      │
│ src/test/resources/cucumber.properties:          cucumber.publish.quiet=true      │
└───────────────────────────────────────────────────────────────────────────────────┘

One more thing - when people have published their first report, they are encouraged to sign in and create a token, and to use this token to publish reports. The Cucumber Reports web site instructs users to do this by defining CUCUMBER_PUBLISH_TOKEN=...... before running Cucumber.

However, as with CUCUMBER_PUBLISH_ENABLED - this has no effect on the JUnit 5 runner. It is possible to activate publishing with a token by adding cucumber.publish.token=..... to src/test/resources/junit-platform.properties, but this is obviously insecure, as the token would typically end up in source control this way.

From what I can tell, the only way to enable publishing with CUCUMBER_PUBLISH_TOKEN is to lean on features offered by the build tool. With Maven, it can be done as shown in https://github.com/cucumber/cucumber-java-skeleton/pull/70.

We need to figure out a way to make this process more discoverable for users. Any suggestions?

Could we tweak the pom.xml of the archetype to set up the profile(s) suggested in cucumber/cucumber-java-skeleton#70?

Could we tweak the pom.xml of the archetype to set up the profile(s) suggested in cucumber/cucumber-java-skeleton#70?

I don't think either change is appropriate. Both are supposed to provide minimal examples of a working project. Cucumber reports is an external service and not really minimal. You may want to maintain a separate project to document Cucumber reports.

I agree it might be useful to have a separate project to illustrate how to configure Maven to publish to Cucumber Reports.

I still think we need to change the banner though, because in its current state it leads users down a dead end. What changes would you suggest we make to the banner @mpkorstanje?

Banner is fine. The change to the archetype and skeleton aren't.

Banner is fine. The change to the archetype and skeleton aren't.

The current banner is fine, or my proposed changed banner is fine @mpkorstanje?

The proposed banner is fine. The change to the archetype and skeleton aren't.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two months if no further activity occurs.

See: https://github.com/cucumber/cucumber-jvm/commit/f507c4cfdd06b6a2cc1473eb7172b05e0b7932d6