BDD Integration

[jbadeau]

Hello,

I have looked into added gherkin support to opentest but i don't think it's a great fit due to missing built in features like:

• ability to call other features
• built in $data support
• store support
• parallelization of steps

It can be done but I think the effort is to high.

I think something closer to https://github.com/getgauge/gauge is a better fit.

I have forked opentest and am now working on an a DSL which better matches opentest

some of the features will include:

• Desktop (eclipse, vscode) and Web (monaco) editor support
• built in support for opentest features
• transpilation to opentest yaml format
• Built in rest and openapi grammers

Regards,

[SuperAlokB]

Thank you for the update - was trying to fit with cucumber but it required a lot of efforts, and it is out os cope for me :(

[jbadeau]

I am currently working on the DSL grammar itself:

https://github.com/jbadeau/opentest/tree/BDD-Integration/bdd

I will be looking at differenty styles to try and find a good balance between flexibility and readability

[adrianth]

Hey Jose, thanks for looking into this. I think it would be best to add BDD (Gherkin) support to OpenTest due to its larger adoption, but, with what I learned so far, it would seem that the Gherkin syntax doesn't support all the features we need. We could probably work around the limitations that you described in the original post, however there are others that I'm not sure how we would handle. For example, how do I describe that I want a particular step to execute on a specific test actor? I'm thinking we might be able to use comments to describe things that we need that are not supported in Gherkin out of the box. Not sure how good that's going to work - I need some time to think through the downsides. In any case, let's define the problem correctly and investigate the best ways to solve it before you spend a lot of time working on integrating a DSL into OpenTest. I would hate for all your effort to be wasted.

[jbadeau]

Hi,

I am not implementing anything yet. I am looking into defining a grammar style that supports openetest features while staying as close to standards like gherkin and gauge.

The idea is that any BDD style language (gherkin, gauge, custom) is transpiled to opentest yaml files with sourcemaps for debugging

I have done similar things here https://github.com/Jose-Badeau/boomslang-core/wiki/User-Documentation

I have also started to document design concerns. For instance, why should a test writer manually define where a segment is executed? Should this not be be a concern of load balancing? Why cant the open-test server or test transpiler do this?

Maybe I should move the concerns to this issue?

[jbadeau]

Oops, accidentally closed the issue

[jbadeau]

I think I now understand what the segments are for. Correct me if I'm wrong but it appears the segments main usecase is not performance, but rather for test orchestration. For example:

1. A book order is posted via a store rest api (segment-1 - actor1)
2. The order is inserted in the stores database and should create an order message to the book supplier
3. The message is checked in the book suppliers system via a rest api (segment2 -actor2)

Is this correct?

If so then maybe a separate dsl for test orchestration is needed.

[adrianth]

You are correct @jbadeau. Test segments execute sequentially form segment 1 to last segment. No work in segment 2 is being started until segment 1 finished execution. This is useful when doing distributed testing involving test actors running on two or more different machines.

[jbadeau]

Ok. I only now read through the docs, it was pretty clear actually 😬.

So I am now focusing on a test grammar and an orchestration grammar. To me the they are a separate concerns. I will of course keep the yaml format.

Give me a few weeks I come up with some proposals

[dmcgill50]

Any update on this feature @jbadeau or @adrianth ? This is something that we would be interested in when it becomes available.

[adrianth]

Hey @dmcgill50, actually I did give this some thought. Initially, it seemed like it would be a very messy exercise to try and represent a test flow as a Gerkin file (as @jbadeau also realized), since Gerkin is missing the language to describe distributed tests, as well as a few other OpenTest features. For example, there's no syntax construct that we can use to include metadata such as what actor should execute a particular step. However I think I have some ideas how to implement it in such a way so that we don't have to muddy up the feature files with useless metadata, but also keep things clean and elegant in order to maximize maintainability of both the feature files and the feature steps. I still have a few more pieces missing in this puzzle, so I need to spend a bit more time on it. Will use this issue to keep you in the loop with where I'm at, but it's not going to be ready any time soon, because it's a pretty involved feature.

[dmcgill50]

@adrianth anything that I can help with this on front?

[adrianth]

@dmcgill50 If you have some experience with BDD already, I might need some help understanding certain concepts. Have you used BDD before? What's your level of familiarity with it?

I"m trying to figure out things like what are all the possibilities to insert a data table in a feature file. I am trying to make sure I don't leave out any use cases when implementing BDD support.

[dmcgill50]

I have. I'd be happy to help test your MVP and then see where the testing takes us. https://cucumber.io/docs/gherkin/reference/ would be a starting point for me in supporting the BDD Syntax.

[jbadeau]

Hi, I don't think cucumber is a good option as it doesn't support includable scenarios like gauge concepts which easily act as segments.

I have added a Gauge example here. It is a very quick and dirty way to demonstrate how integration could be done:

• generate a test yaml file for each scenario (demonstrated)
• read in a gauge file and map to org.getopentest.testdef.TestDefinition (would prob need to use the gauge LSP infra)
• call the opentest api directly like the angularjs sessionCtrl and actor. This is more flexible but more complex. If you want to go this way then I would suggest a lil planning session to flesh out mvp features

In general a bdd approach is nice because it:

• hides technical details about the test like actors and segments
• allows for reusable segments and steps
• allows both technical grammar like opentest or business domain

When you decide u want a generator approach then I will add an API to make working with the test structure easier but this should provide enough to get started. Things to add:

• java or typescript or even a web api to create test files
• typed support for api, mobile and web
• configurable tank output location
• @adrianth publish the actor jars to maven or make an api jars. I had to copy the class and make them valid beans, Would be nice the selenium and web apis were public too.

[dieperei]

Hello guys, any update on this feature?

Thanks in advance

[dmcgill50]

It looks like the POC has been moved as that link is broken.

Any thoughts Adrian?

On Wed, Jan 22, 2020 at 11:00 AM Diego Pereira [email protected] wrote:

Hello guys, any update on this feature?

Thanks in advance

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mcdcorp/opentest/issues/24?email_source=notifications&email_token=AABYHQETDWO2UFGDYVZUWFLQ7B3SPA5CNFSM4FQU3ZW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJUKRDY#issuecomment-577284239, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABYHQCEXHOD5QRW63X3PRTQ7B3SPANCNFSM4FQU3ZWQ .

-- David McGill (405) 628-5297

[adrianth]

I believe the work stream started by @jbadeau has been abandoned.

OpenTest will eventually support the Gherkin syntax, eventually, but there are some other updates that must happen before we can properly implement that feature. One other thing to note is that the Gherkin syntax doesn't have the ability of expressing a distributed test that leverages multiple test actors, potentially running on different machines, so that will be a limitation when writing tests BDD-style.

[dmcgill50]

I think that limitation is sufficient for the benefit. Please let me know if there are things that can be worked on to allow for this new BDD style to be supported.

[adrianth]

@dmcgill50 Realistically, BDD will not be available very soon, because it requires a non-trivial amount of work:

• designing the workflow
• designing the coding conventions
• modifications to the core framework (biggest effort of all)
• actual implementation work

Thanks for the offer and I will involve you as soon as there's something that can be worked on collaboratively. Is there a big appetite for the BDD approach in your organization?