common
common copied to clipboard
cucumber
Remove JSON/HTML/Pretty/JUnit XML formatters from all implementations
The introduction of a new Rule keyword (https://github.com/cucumber/cucumber/pull/414) and the collapse of Scenario and Scenario Outline (https://github.com/cucumber/cucumber/pull/353) changes the Gherkin AST and API considerably.
Existing JSON/HTML/Pretty/JUnit XML formatters implemented natively in all implementations will need significant change to cope with the new AST. It is obviously possible to update them all, but they are all pretty complicated (and therefore probably buggy), and it will be hard work.
I think it would be much better to implement these as standalone command line tools (probably in Go, for portability) that read an event stream from STDIN (encoded as protobuf and/or JSON) and produce reports to STDOUT or the file system (when multiple files must be written, such as with HTML and JUnit XML).
These formatters could be hooked into existing Cucumbers, and when implementations start using the pickle runner (cucumber backend), they could be hooked up there instead.
I'd like to get completely rid of the JSON formatter. It's obsoleted by the event stream. Third party tools that are currently reading the JSON format output must be modified. This will cause some friction, and it will take some time, but the end result will be much better for everyone:
- A consistent, well defined schema (as opposed to cucumber.json which is a mess)
- Real-time reporting, so tools can update the report as scenarios are running
- Less memory consumption, as there is no need to load all the results in memory (thanks to the event stream)
The other reports should be reimplemented. Another obvious benefit is that we only need to maintain a single implementation of formatters.
Thoughts and feedback welcome @cucumber/committers.
The litmus test for this would be to verify that no code in Cucumber depends on gherkin.ast.
I'm really keen to see this happen. I think standardising on an event protocol and having one set of formatters to maintain would be an excellent step forward.
What I think we need to do is work out what the baby steps are to get us there. What are the bites to eat the elephant. I wonder if we need to have a conversation with at least @brasmusson @charlierudolph @mpkorstanje and @aslakhellesoy and try to mind-map it all to figure out a sensible route to the goal?
What’s worked well recently is to build a standalone library (gherkin, cucumber-expressions, tag-expressions, datatables) that is tested independently. When we feel it’s ready to be integrated, each implementation does that at its own pace.
When that happens we usually find small issues, which are easily fixed, then integrate and ship a new major version of Cucumber.
Personally I would like to start with a HTML formatter.
I know @nfisher is working on gherkin-pretty in #400. @nfisher how do you feel about making this tool able to consume result events too, so that it can colour/highlight output and become the new portable pretty formatter?
Any volunteers for JUnit XML?
The more consumer-driven the better!
... probably in Go, for portability ...
This isn't relevant for Cucumber-CPP because it doesn't contain such a pretty formatter. But I'd thought I'd mention this anyway: Go is still nowhere near as portable as C and C++. I'm using this on some ARM based embedded targets where I'm happy that we've recently dropped the last platform that didn't have C++11 support. Even C++11 support isn't completely usable on some targets. So a much more recent thing like Go isn't even a conceivable option in these kind of spaces as platform vendors definitely won't be adding a cross compiler for that anytime soon (where soon means in the next 5 years).
Luckily I have so far been able to run with the Cucumber-CPP "core" on the target platform and connecting Cucumber to it via the wire protocol from a host machine. So as long as such splits remain possible the problems remain manageable.
BTW I think “message protocol” is a better name. Some messages will be events, but others will be commands. I started a new cucumber/messages project yesterday that uses protobuf rather than JSON/JSON Schema - will push soon. This will make platform interop a lot easier, and also reduce the amount of message code we have to write because protoc will generate message structs for any language.
@muggenhor I’d assume 99.9% of Cucumber users work on Linux/MacOS/Windows, where Go works fine. C/C++ competency is scarce compared to Go, and much harder to do well.
In this context Go seems the best choice to me.
@aslakhellesoy I've stalled development as I wanted to get some input on the open PR. Namely I implemented the fmt.Stringer interface on the AST types as it is the quickest way but perhaps not the preferred way to implement it.
Lessons learned from implementing parallel execution in cucumber-jvm:
- The protocol should describe a weak ordering in which events may arrive
- The events should contain enough information to allow the execution of pickles to be interleaved.
- Formatter implementations should take interleaved execution into account from the start (it results in a significantly different implementation compared to serial processing).
I don't have much else to contribute at this point.
I think we can use fuzz testing here. Send a bunch of events (loaded from file) and expect a certain output. Run it many times with fuzzing - shuffling the events according to weak ordering.
What’s worked well recently is to build a standalone library (gherkin, cucumber-expressions, tag-expressions, datatables) that is tested independently. When we feel it’s ready to be integrated, each implementation does that at its own pace.
Sounds good to me.
I think an interesting piece of this work will be doing the contract testing so we ensure the producer(s) and consumers of these messages are using a consistent protocol. We had a crack at that before but it felt pretty clunky.
It's also worth mentioning https://github.com/cucumber/cucumber/tree/gui-add-initial-spike which was the original inspiration for the event protocol as implemented in cucumber-js. That makes an excellent consumer for testing these events and I'd love to see it become a thing.
I think an interesting piece of this work will be doing the contract testing so we ensure the producer(s) and consumers of these messages are using a consistent protocol. We had a crack at that before but it felt pretty clunky.
With Protocol Buffers will don't need to test that the messages themselves are well-formed. Protobuf already ensures that.
All we need to test is that a consumer can consume messages in the order they are received.
There is an embryonic start of Ruby adapter inside dots.rb. We should move this over to cucumber-ruby I think.
The runtime will have a StreamFactory injected, and will ask it for a stream to write messages to (could come from launching an executable, or connecting to a socket). We will have some tests that verify the correct messages are written to this stream.
It may be worth considering a framework level flag to create individual reports of each scenario, rather than a larger document with all scenarios. The eventbus would handle this nicely I’d think. It may be neat to explore the “progressional” aspects of cucumber while its running. “Hot off the press” attitude! Cheers!
Since my Cluecumber Report Plugin needs Cucumber JSON as input, I would like to help with this matter. However, I don't fully understand what needs to be done. Are there any more concrete next steps I can tackle (I am mainly a Java developer with a little bit of go experience)?
@aslakhellesoy is this still a thing? The JSON formatter is in maintenance mode I think the new HTML formatter based on react has been released in ruby, java and javascript I am not sure we could really remove junit formatters for the same reasons as the JSON formatter And I guess pretty formatters are pretty tied to their implementation
WDYT?
Closing due to inactivity. We can re-open individual tickets for specific formatters as we decide to tackle them.