I would like to see Cucumber as supported framework for e2e testing.

[asnov]

I would like to see Cucumber as supported framework for e2e testing. Thank you, guys!

[samhatoum]

I would also like to see this.

I am one of the maintainers of Chimp which allows you to run Cucumber with end-to-end tests. I'd be happy to collaborate with you @ArtemGovorov to get this working

Cucumber.js has a command-line runner and it outputs a JSON report file. I'm guessing we'd need to add it as a runner somehow and transform the output to something that wallaby can parse.

Any pointers on how I can start to mess with it?

[ArtemGovorov]

Hi @samhatoum, Chimp looks awesome, thanks for sharing.

Regarding the integration, I'm not familiar much with how Chimp/Cucumber works, so I'll try to describe what would be required from wallaby side.

A new runner most certainly is required (and that part of the core is not yet available for external extending unfortunately).

Regarding the output, wallaby.js needs quite a few things:

• Apart from the runner, the JavaScript that Chimp/Cucumber executes has to instrumented by wallaby core (to collect the code coverage data).
• Wallaby also needs to hook into the testing framework (by adding a reporter), so that it knows when a test/suite is completed, able to filter what tests to skip/execute, etc.

By instrumenting files and adding the reporter, wallaby.js is able to get the information it needs. With our PhantomJs runner, we create a web socket connection using which wallaby.js sends the collected data to its core (server).

[samhatoum]

Thank you @ArtemGovorov

I'll try to improvise on each point to say how I can see it being done:

• The instrumentation can be done prior to a Cucumber.js execution
• When Cucumber.js is complete, it generates a json file, which we can transform to a wallaby-compatible report
• The next Cucumber.js run can be passed only the filtered features to run

Typically Cucumber.js runs in a node environment (it's also possible to run on the client, but not a typical practice).

It sounds like the reporter is where the work would be. Can I help in any way?

[ArtemGovorov]

The instrumentation can be done prior to a Cucumber.js execution

Yes.

When Cucumber.js is complete, it generates a json file, which we can transform to a wallaby-compatible report

This is the hard part. Wallaby needs quite a few data bits from the test execution host, ideally in real-time just as tests are executing.

Thanks, I may need some help with the reporter, I first should find some time and have a deeper look into Cucumber.js to better understand how it works, how to make it send what wallaby needs and generally what's the best approach of integrating with it.

[samhatoum]

This is the hard part. Wallaby needs quite a few data bits from the test execution host, ideally in real-time just as tests are executing.

Look no further than the StepResult event. This can give you realtime results as step are executed happen.

To use it with Wallaby.js, you'd have to create a file like step-result.js file, and put this in it:

module.exports = function() {
  this.StepResult(function(event){
    var stepResult = event.getPayloadItem('stepResult');
    var step = stepResult.getStep();
    //console.log(stepResult); // be sure to check this out
    //console.log(step);  // be sure to check this out

    console.log('\n - - - - - STEP RESULT - - - - - ');
    console.log('stepResult.isFailed - -', stepResult.isFailed());
    console.log('stepResult.isPending - -', stepResult.isPending());
    console.log('stepResult.isSkipped - -', stepResult.isSkipped());
    console.log('stepResult.isSuccessful - -', stepResult.isSuccessful());
    console.log('stepResult.isUndefined - -', stepResult.isUndefined());
    console.log('stepResult.getDuration - -', stepResult.getDuration());

    console.log('\n - - - - - STEP - - - - - ');
    console.log('step.setPreviousStep - -', step.setPreviousStep());
    console.log('step.isHidden - -', step.isHidden());
    console.log('step.isOutlineStep - -', step.isOutlineStep());
    console.log('step.getKeyword - -', step.getKeyword());
    console.log('step.getName - -', step.getName());
    console.log('step.hasUri - -', step.hasUri());
    console.log('step.getUri - -', step.getUri());
    console.log('step.getLine - -', step.getLine());
    console.log('step.getPreviousStep - -', step.getPreviousStep());
    console.log('step.hasPreviousStep - -', step.hasPreviousStep());
    console.log('step.getAttachment - -', step.getAttachment());
    console.log('step.getAttachmentContents - -', step.getAttachmentContents());
    console.log('step.getDocString - -', step.getDocString());
    console.log('step.getDataTable - -', step.getDataTable());
    console.log('step.hasAttachment - -', step.hasAttachment());
    console.log('step.hasDocString - -', step.hasDocString());
    console.log('step.hasDataTable - -', step.hasDataTable());
    console.log('step.ensureDataTableIsAttached - -', step.ensureDataTableIsAttached());
    console.log('step.isOutcomeStep - -', step.isOutcomeStep());
    console.log('step.isEventStep - -', step.isEventStep());
    console.log('step.hasOutcomeStepKeyword - -', step.hasOutcomeStepKeyword());
    console.log('step.hasEventStepKeyword - -', step.hasEventStepKeyword());
    console.log('step.isRepeatingOutcomeStep - -', step.isRepeatingOutcomeStep());
    console.log('step.isRepeatingEventStep - -', step.isRepeatingEventStep());
    console.log('step.hasRepeatStepKeyword - -', step.hasRepeatStepKeyword());
    console.log('step.isPrecededByOutcomeStep - -', step.isPrecededByOutcomeStep());
    console.log('step.isPrecededByEventStep - -', step.isPrecededByEventStep());
    console.log('step.getStepDefinition - -', step.getStepDefinition());
  });
}

And then when you run cucumberjs from Wallaby, just pass it an additional file using -r <path to>/step-result.js.

That should do it!

[ArtemGovorov]

Thanks Sam, will look into the StepResult.

[timReynolds]

Plus one comments are super annoying but I'd be interested in seeing Cucumber support for wallaby.

[eggyal]

I'd also like to see support for Cucumber, or else to understand how one can get Wallaby working with Yadda (which ostensibly overlays Gherkin-style BDD support atop one's test framework, including Jasmine and Mocha)—any thoughts?

[ArtemGovorov]

@eggyal if Yadda is based on Cucumber, then this issue is definitely a prerequisite before supporting it.

[eggyal]

@ArtemGovorov: No, it doesn't depend on Cucumber. AIUI, your existing test framework (Jasmine, Mocha, whatever) uses Yadda to parse Gherkin-like feature files in order to identify which test functions from your spec files should be run (and with what arguments).

The problem I've had is that it's only really practical with a single entry-point, and I think this would break Wallaby's caching?

[ArtemGovorov]

@eggyal Got it, thanks.

it's only really practical with a single-entry point, and I think this would break Wallaby's caching?

Not sure about caching, but can definitely see that a few of main wallaby features, such as running tests in parallel and incrementally executing tests, will not work if the framework expects a single-entry point for executing specs.

[eggyal]

Yes, one would invoke mocha ./yadda-entry-point.js or whatever.

So I guess Yadda's resolution of which spec functions to invoke would have to take place within Wallaby... is there a plugin API or somesuch that I can harness?

[ArtemGovorov]

@eggyal There's no (public) API at the moment to perform such integration.

[chrisns]

In terms of the yadda stuff I put this example together so at least the wallaby hackery/work can live somewhere. https://github.com/chrisns/yadda/tree/wallaby/examples/mocha-wallaby I was trying to see if I could fool wallaby into working, turns out its too clever :) Tests run beautifully the first time, I was super excited, in fact I changed the feature file a couple of times and it was all working, then it for no discernible reason stopped and started showing various irrelevant errors. Obviously coverage + inline beautiful wallaby goodness in the .feature file are probably a little far fetched for now at least, but it would be wicked to be able to execute the tests.

[celador]

@ArtemGovorov Any idea for when/if this might happen? or when "the core may be available for external extending". Wallaby is really nice, but Chimp.js is more full featured. I bought Wallaby, and will likely use it for writing js unit tests, but having Chimp.js/Cucumber support would be a killer feature. Wallaby will be a hard sell to my team without Chimp/Cucumber support. I work on a team with 40+ developers and it would be an easy sell to the team if it had Chimp support. We'd certainly purchase a license for each developer if we could have Chimp(Chrome, Gherkin, Webdriver.io API) + Wallaby(super fast testing and code coverage in the IDE) . Please, please, please work with @samhatoum to see if you can get this working.

[ArtemGovorov]

@celador I have done some estimates and looked into Cucumber/Chimp to better understand our options back in Feb, there are a few things that need to be implemented in wallaby.js core before implementing the support. Some of them are pretty big. Nevertheless, I'd like to add Chimp/Cucumber support at some point, but can't make any promises when it's going to happen.

[samhatoum]

So Chimp is a test runner, and Wallaby is a test runner. A test runner running a test runner is a bad idea!

Saying that, Chimp has a session manager that makes end-to-end a pleasurable experience by reusing open sessions. There may be some value in extracting the session manager into a separate module, which can then be imported into a Wallaby + Cucumber.js configuration - but I think I'll only be able to do that once we see the basic Wallaby + Cucumber combo.

@ArtemGovorov let me know if you need anything when you finally get around to doing this

[samhatoum]

Hey @ArtemGovorov. I know how busy life gets and how product backlogs work. Just wondering if you have any rough outlook on when you might start attacking Cucumber.js support. And as always, here to help if you need it as I've done some major digging into Cucumber.js.

Thanks!

[ArtemGovorov]

Hey @samhatoum, can't share any plans about the feature support yet. The team is busy at the moment rushing towards releasing this big feature. After releasing that one and clearing up a few smaller long wanted/waiting ones, we'll revisit the backlog, and of course will post some update here if anything changes.

[samhatoum]

@ArtemGovorov that looks incredible!! I love it and can't wait to try it.

[eggyal]

Wallaby App is indeed very cool, but the benefit it actually provides for us (over monitoring test results in-editor) doesn't come anywhere close to what we'd get from having Cucumber support.

I appreciate that there are other priorities, but it nevertheless saddens me that (15 months after it was first requested) there's still no indication this feature is even scheduled to be worked upon any time soon. Perhaps I've just become too used to how rapidly open-source tools iterate, or else how easy it is to add custom functionality where there are external APIs for plugins/modules!

I'll keep praying for this feature to appear sooooooooon!

[eggyal]

Maybe https://github.com/cucumber/microcuke (an official, minimal—only 500 SLOC, JS reference implementation) could be useful?

[tedvanderveen]

@ArtemGovorov can you please share status on this enhancement request?

[ArtemGovorov]

@tedvanderveen The status hasn't really changed, but I'll post an update here when it does.

[jfstephe]

For me, test coverage is not a great measure of quality (not that people are necessarily saying it is) but perhaps at best an indication of the quality of code. All the arguments about this aside, if you are running cucumber tests and non-cuke tests if you can only report the test coverage on part of the tests you are running that makes this less than useful for me. Was quite excited about this tool until I saw this was not supported.

ETA on this feature? 1 week? 3 months? 6 months? 12 months? Never? Can you give us a ball-park idea?

[ArtemGovorov]

@jfstephe

ETA on this feature? 1 week? 3 months? 6 months? 12 months? Never? Can you give us a ball-park idea?

Unfortunately I can't share ETA at the moment. It is highly unlikely that we add the support within the next 3 months.

[samhatoum]

@ArtemGovorov @charlierudolph is the maintainer of CucumberJS. I'm hoping this intro could accelerate this ticket :)

[eggyal]

FWIW, I see that #1144 was linked in the context of Cucumber being for E2E testing. That's not really correct, at all. Cucumber is merely an engine for running Gherkin files, which are themselves a way of specifying tests in natural language—those tests could be unit tests, integration tests, E2E tests or indeed anything at all.

Certainly it's true that the overhead of writing tests in natural language is typically reserved for cases where they are reviewed by a non-technical audience, e.g. acceptance tests (which are usually only E2E)—but there is nothing that requires that to be the case.

It may be that this issue was lumped together with E2E testing frameworks for brevity, but I thought it worth setting the record straight lest a misunderstanding or incorrect assumption have caused it to be given lower priority than it might otherwise have received.

[samhatoum]

I'm hawking this issue as Wallaby support for CucumberJS would make me jump for joy.

So I second the comment from @eggyal. At Xolv.io we don't use Cucumber for e2e testing at all, we use it to do drive development of the system. If wallaby is to run tests within the browser/node context, this is the same use case for Cucumber as it is for mocha/jasmine/tape. From wallaby's perspective, Cucumber should be treated just as another test framework.

I also want to reiterate that Chimp has nothing to do with this implementation.

More interestingly, a new development in CucumberJS 3.0 is that it now has an event-protocol for reporters and other plugins.

See here: https://docs.cucumber.io/event-protocol/

and here: https://github.com/cucumber/cucumber-js/blob/master/src/formatter/event_protocol_formatter.js

@ArtemGovorov I think this might give you the realtime-ness you're looking for in a decoupled fashion

[FlippieCoetser]

@ArtemGovorov any new feedback regarding this issue? WallabyJS is great having it support CucumberJS would be even better.