Message based cucumber-junit-formatter

[aslakhellesoy]

We need a cucumber-junit-formatter executable that reads messages from STDIN and writes JUnit XML to STDOUT. Any Cucumber implementation with a message formatter can then pipe generated messages to this formatter to generate JUnit XML.

Older Cucumbers with built-in JUnit formatters will have those removed when they support messages and people will have to use the cucumber-junit-formatter executable instead. The existing ones are:

• JUnitFormatter.java
  • test data
• junit.rb
  • test data

As pointed out in https://github.com/cucumber/cucumber-ruby/issues/1384 the generated XML should contain the steps (like the Java implementation does).

In order to be portable, cucumber-junit-formatter should be implemented in Go. Look at demo-formatter/go for inspiration.

[vincent-psarga]

I don't think it should be a stand alone tool. I think there should be a cucumber/junit-formatter in the mono-repo, written in all languages currently supported (Java, Ruby and JS). We could of course take advantage of the messages portability to have a common set of tests. JUnit is one of the most widely used output format, and I definitely think it should built-in all cucumber releases, without having to add an extra tool

[mpkorstanje]

Might be of interest that the JUnit Team is currently working on a new format too support the features in JUnit 5. I would wait too see how that turns out before we spend too much time implementing a structural solution based on the JUnit 4 format.

https://github.com/junit-team/junit5/issues/373

[mpkorstanje]

Also worth nothing that the JUnit formatter works just fine, even with the message protocol. So I don't see a need to remove it. I can see a use case for implementing it as a polyglot component with a common acceptance test suite.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[aslakhellesoy]

@vincent-psarga is right. It shouldn't be a standalone tool, but a library that can be used with all the supported Cucumber implementations.

Haxe is nice, but it lacks support for Ruby and Go, which are two of the languages we support. So I'm afraid we have to go with the one-implementation-per-language approach several other libraries in the monorepo uses.

[aslakhellesoy]

That sounds interesting. If it turns out to work well it might reduce duplicate efforts quite a bit!

[mpkorstanje]

@denim2x your formatter is using the events from Cucumber JVM. However Aslak is looking for an implementation that uses the cucumber messages format. It's set of messages defined in protobuf.

See: https://github.com/cucumber/cucumber/tree/master/messages

[mpkorstanje]

Also worth noting that while a Kotlin implementation would be executable on the JVM, we can not use it in Cucumber JVM. To avoid conflicts with the system under test Cucumber should pull in as few third party dependencies as possible (i.e. zero). A Kotlin implementation would add a dependency on the Kotlin Standard Library.

[aslakhellesoy]

@denim2x where can we follow the development of this transpiler?

[davidjgoss]

@aslakhellesoy I'm interested in working on this. Would it be okay implemented in JavaScript (using messages, not coupled to cucumber-js) in similar way to html-formatter?

[aslakhellesoy]

Yes that would be amazing @davidjgoss!

[davidjgoss]

First pass at a JavaScript implementation here: https://github.com/davidjgoss/cucumberjs-junit-formatter

Here's an example from a test of what it outputs: https://github.com/davidjgoss/cucumberjs-junit-formatter/blob/main/test/snapshots/integration.test.ts.md

It's wrapped as a cucumber-js Formatter class, but doesn't use any of the old formatter helpers - just consumes envelopes and uses gherkin query and cucumber query.

We're now talking about having it live in the monorepo as a generic envelopes -> junit stream with a thin binding in cucumber-js (similar to the html formatter). We could then port other platforms' junit formatters here and share a test suite driven from the compatibility kit (which would ideally need a few more cases e.g. pending, undefined, ambiguous).

Things to maybe iron out:

• As @mpkorstanje points out, for each <testcase> the combination of classname and name should be unique. Currently these are populated with pickle.uri and pickle.name respectively, but the latter isn't guaranteed to be unique (e.g. with scenario outlines) so need to add some kind of qualifier.

[mpkorstanje]

The code that makes the names unique can be found here:

https://github.com/cucumber/cucumber-jvm/blob/main/core/src/main/java/io/cucumber/core/plugin/JUnitFormatter.java#L208

But probably best not to copy that as it assumes scenarios are executed in order. This is not guaranteed for messages.

If not unique, I would suggest numbering the nodes in a feature file and using the path to the pickle as a unique prefix.

1. Rule
  1.1 Scenario
     1.1.1 Examples Section
         1.1.1.1 Example 1
         1.1.1.2 Example 2
         1.1.1.3 Example 3
     1.1.2 Examples Section
         1.1.2.1 Example 1
         1.1.2.2 Example 2
         1.1.2.3 Example 3
2. Rule
  2.1 Scenario
 ....

[luke-hill]

@mpkorstanje if doing that wouldn't it be easier to use line number? As that's already in the AST right? Or am I getting confused?

[mpkorstanje]

Yes. Would be easier, but the reason why it is there would be less obvious.

[aurelien-reeves]

As @mpkorstanje points out, for each <testcase> the combination of classname and name should be unique. Currently these are populated with pickle.uri and pickle.name respectively, but the latter isn't guaranteed to be unique (e.g. with scenario outlines) so need to add some kind of qualifier.

If it is the combination of classname and name which should be unique, then pickle.uri and pickle.name should work. Am I missing something?

[davidjgoss]

@aurelien-reeves pickles from the same feature file could have the same name if:

• You have a scenario outline with examples (unless the name includes a parameter)
• People have just given several scenarios exactly the same name (~bad~, not invalid Gherkin)

[aurelien-reeves]

Oh, ok.

What would you think of adding the line number in the URI, as an anchor?

[mpkorstanje]

Okay. Other suggestion then. We can do what ever JUnit does when it encounters duplicate.

I've seen JUnit do something like this (I think, would have to double check)

ExampleTest
  ParameterizedTest [0]
  ParameterizedTest [1]
  ParameterizedTest [2]
  ParameterizedTest [3]

[luke-hill]

@aurelien-reeves pickles in the same feature file could have the same name if:

• You have a scenario outline with examples (unless the name includes a parameter)
• People have just given several scenarios exactly the same name (bad, not invalid Gherkin)

I have the following gherkin frequently (Which I don't consider bad)

Feature: Foo
  Rule: English language
    Scenario: Can do Foo

  Rule: Welsh language
    Scenario: Can do Foo

[davidjgoss]

@luke-hill true - I forget that the rule name doesn't contribute to the pickle name

[aurelien-reeves]

Okay. Other suggestion then. We can do what ever JUnit does when it encounters duplicate.

I've seen JUnit do something like this (I think, would have to double check)

ExampleTest
  ParameterizedTest [0]
  ParameterizedTest [1]
  ParameterizedTest [2]
  ParameterizedTest [3]

Only for duplicates right?

[davidjgoss]

Only for duplicates right?

Yep! Here's a PR that does that https://github.com/davidjgoss/cucumberjs-junit-formatter/pull/1

[mpkorstanje]

Nasty case:

Scenario: A
Scenario: A
Scenario: A[1]

[davidjgoss]

@mpkorstanje oh come on, that's just gratuitous! 😆

[davidjgoss]

We have a PR from the community to add a JUnit formatter to cucumber-js: https://github.com/cucumber/cucumber-js/pull/2121

It doesn't consume pure messages yet but I think it's a better implementation than my POC from last year in other ways. I think it would be worth accepting it as a step towards this goal - we can spin it out into the common repo when we're ready. In terms of the output (e.g. how it expresses failures, what it puts in <system-out>) it seems like it should follow what cucumber-jvm does in the absence of any other standard.

Any thoughts?

[mpkorstanje]

I agree. Note that the Cucumber JVM implementation is tricky and the JUnit XML a very poorly defined standard. I'll add my remarks to the other PR.

[mpkorstanje]

I've found documentation for the JUnit XML format:

• https://llg.cubic.org/docs/junit/
• https://github.com/junit-team/junit5/blob/43638eb6a870e0d6c49224053dfeb39dcf0ef33f/platform-tests/src/test/resources/jenkins-junit.xsd
• https://github.com/junit-team/junit5/issues/86
• https://stackoverflow.com/questions/4922867/what-is-the-junit-xml-format-specification-that-hudson-supports

[mpkorstanje]

Java is done. Next step would be to extract the formatter added in https://github.com/cucumber/cucumber-js/pull/2121