cucumber-jvm

cucumber-jvm copied to clipboard

Currently, we can only access Scenario level information in @BeforeStep & @AfterStep, it would be really great having step level information like, step name, line no, etc.

This will help to add implementation like, common step level log.

A common cause for this request is the need to create a custom report. Hooks however are designed to do things before and after a step and not suitable for generating reports. Instead you may want to examine the Plugin system. You can find a recent blog post here and you can find examples of existing plugin here (they're still called formatters internally).

However should you still need this information in a before step hook please feel free to send a PR. For the end user this should look something like the example below.

@BeforeStep
public void before_step(Scenario scenario, Step step){

}

Maybe Scenario could contain a getCurrentStep() method that way it won't be needed to enhance the existing annotations. See also #1713 acess to the Glue Classes from a before/after step would be also very usefull.

This will help to add implementation like, common step level log.

as @darsh9292 mentioned logging here, one might want to add information from the current step to the MDC, beside that, it sounds obvious that a before/after hook can access the object it is a hook for (here a step) .

one might want to add information from the current step to the MDC,

Again, you may want to examine the Plugin system. Step hooks hooks are only invoked around regular steps. Not around hooks so your context would be incomplete if you tried to use step hooks for this.

Maybe Scenario could contain a getCurrentStep() method that way it won't be needed to enhance the existing annotations.

Such a method would have to return null when accessed from a @Before or @After hook as there is no current step in that context. So while more convenient to implement, it would be harder to document and use.

Just wanted to give a hint why @darsh9292 might has use for this. Returning null if there is no actual current step seems appropriate to me. A plug-in might be too broad and does not has access to the current glue, but I can only guess whats the real use-case is.

Hi there,

I'd like to have this feature if possible too. Reason been I'd like to include the details of the step name and accompanying line that failed thus making the test authors life easier when triaging the test report 😀. This could especially become useful when a particular step is executed more than once then step.getLine() in particular becomes very useful.

Best Regards,

@bhreinb you shouldn't use hooks for reporting. Please have a look at the plugin system.

Hi @mpkorstanje

Thanks for the response. I can have a look at the plugin system as you say. I'm curious as to why you can't use hooks for reporting when the suggested method implementation is like so

@BeforeStep
public void before_step(Scenario scenario, Step step){

}

Ideally the step object would provide the necessary detail of the step for example the name & line of it. The scenario object already provides an API to embed a mime type to the JSON report. I guess why couldn't that work?

Hooks can be used to add additional information to the report by using scenario.attach/scenario.log. Creating the actual report is cross cutting concern that shouldn't be mixed with glue code. You'll find that all information you need is already available in the plugin system.

The suggestion for me was to capture the step information at that phase and when appropriate during test step execution (not every time it would be added) and add it to the report JSON via scenario.attach/scenario.log. I wasn't suggesting the creation of a new report, sorry if i wasn't clear. I find it strange that I have to add a plugin for such a case given I can do something similar via Before or After hook.

The step information including any output attached or logged is already available in the json formatters output. I'm not sure what you are looking for then.

    "elements": [
      {
        "line": 4,
        "name": "A Calculator",
        "description": "",
        "type": "background",
        "keyword": "Background",
        "steps": [
          {
            "output": [
              "Main calculator turn on!"
            ],
            "result": {
              "duration": 1389000,
              "status": "passed"
            },
            "line": 5,
            "name": "a calculator I just turned on",
            "match": {
              "location": "io.cucumber.examples.java.RpnCalculatorSteps.a_calculator_I_just_turned_on()"
            },
            "keyword": "Given "
          }
        ]
      },

So yes the json formatter has all that. Generally the json formatter output is passed onto a plugin per say to generate the HTML report (here is an example http://damianszczepanik.github.io/cucumber-html-reports/overview-features.html). I think that would be a lot of users usage of the json formatter output namely to generate the html report.

The html report discards the step line per say (as it's probably noise in the report). For me I want an ability to include the step line via a custom attachment when needs be. Namely I would be using API

public void embed(byte[] data, String mediaType, String name)

to add an embedding to the JSON report to say

This step name "Blah" failed at line number "15". (convoluted example but think you get gist here)

So the request is to be able to get access to the step information within the context of the cucumber test runner for some "custom" usage. My example would benefit in terms of quicker test diagnostics (I'd suspect other users would have different usages of step information though). Hope I convinced you enough that it is something worth adding 😉

The information is already in the json. If you want it included into a html report you should petition the author of cucumber-html-reports to include this information.

Their is no the way the author would make a change as vast as that to facilitate this use case. In any case the topic has diverged. The request was to have access to the step information in the context of the cucumber runner. The BeforeStep and AfterStep is already there. I guess why can't it be extended to include that information? Other users have requested this per the thread above.

Yes, I would agree with @bhreinb , why before_step can not be extended with additional information on a step, where its job is on step level. Then we should not inject the Scenario object as well.

My purpose in raising this request was, I want this step log information on the console log just for additional log information. Also, sometimes I faced, the runner goes into the infinite mode, but I don't know where it was exactly stopped executing.

This will help in case we will not have a final report at all.

Just to be clear, I'll still accept a pull request to add this to the step hooks.

https://github.com/cucumber/cucumber-jvm/issues/1805#issuecomment-542076721

However should you still need this information in a before step hook please feel free to send a PR. For the end user this should look something like the example below.

@BeforeStep
public void before_step(Scenario scenario, Step step){

}

Even I am trying to log the step level information in to extent reports. An method signature like this would be really helpful.

@BeforeStep public void before_step(Scenario scenario, Step step){

}

I'm looking for something similar, however I don't need it for logging, but to do something after each @Given step. I'll try to come up with a pull request...

Any update on this?

`@AfterStep public void before_step(Scenario scenario, Step step){

}`

I'm still not convinced why we should add this given that the plugin API exists.

Is it just that people find the plugin API hard to use?

Is there a complete description somewhere what event contain what information and when they are are and so on for the plugin API? Maybe that's not always clear, I personally just look at existing plugins and start a debugger to see whats going really on but if you have found your way its very powerful but have the following limitations:

1. You need to write extra code on a different API that has much more "power" that one probably need for this simple case (access additional information about the Step the annotation is targeting)
2. You need to "glue" somewhere your Plugin code with your step-code most likely via static fields as (correct me if I'm wrong) plugins can't participate in the DI mechanism as they are instantiated by Cucumber in a very early stage
3. You need to make sure your plugin is actually included in the run
4. Hooks can be contained to only run on given tags for example, that's of course possible for a plugin but requires extra handling if placed in a plugin

on the other hand the proposed style:

@BeforeStep
public void before_step(Scenario scenario, Step step){

}

makes it clear what happens here (no side-band communication) and is something that one intuitively would expect, why should a before-stepp hook should not know about the step?

I agree with @laeubi on the ease of use the additional parameter would bring.

I must admit I never really looked at the plugin API, but in my particular case it seemed a lot of overkill when all I want is to know if the BeforeStep is for a Given or another type of step. (Of course, setting up a pull request and having it hang for half a year isn't exactly much better... 🤷)

Hi there,

I agree with @laeubi & @BenVercammen on the ease of the use the additional parameter would give to users of cucumber.

In certain circumstances I would like the details of the step information within the glue context for some custom usage. I do use the plugin system for other things but for this usage it does seem complete overkill. Looking at a plugin example

https://github.com/cucumber/cucumber-jvm/blob/main/core/src/main/java/io/cucumber/core/plugin/JsonFormatter.java

for me to get the test step information I would have to add a new class to my project implementing the EventListener interface. I would only be implementing

publisher.registerHandlerFor(TestStepStarted.class, this::handleTestStepStarted);

As @laeubi mentioned I would have to add this new class as a dependency plugin when the runner executes a gherkin script. And lastly then I'd had to make the context of the plugin available to glue code in order for me to consume it for my usage. All of the above is possible to do but it makes things much harder than it should be.

Note I was also looking to taking advantage of the PR @BenVercammen submitted. It's definitely something myself and my team would be taking advantage of assuming it gets merged.

I would like the details of the step information within the glue context for some custom usage.

Could you explain this exact usecase?

We test in multiple environment types such as alpha, beta & production environments to list some. Depending on the environment type some product features is on/off. The glue code evaluates this at runtime. Accordingly we have test steps that "execute" based on the environment type. Our glue code looks something like this (I provide a simplified example for the purposes of explaining the use case):

public void executeWebServiceRequest(final String featureName,
                                     final String jsonPayload)
{

   new FeatureEvaluator(featureName).run() -> {

        //Implementation code for the step, i.e execute web service request...

   });

}

Essentially featureName is an optional parameter. If the glue code receives nothing for the optional parameter then the implementation code would be executed as per normal. However if the glue code receives something we check whether the feature is on for the environment type the tests are running on. The run method decides whether to execute the implementation code based on evaluation of the featureName. The step will pass if the featureName evaluates to off, in that it won't execute the implementation logic in the glue code and move to the next gherkin step within the test script, which is what we want. Conversely, the implementation code gets executed if the featureName is evaluated as on.

Their probably will be strong opinions on the usage of cucumber here admittedly. However for us this works adequately as it allows us to use the same test scripts across multiple environment types plus allowing us to test with features turned on/off too. In addition, our glue code is generic in the sense it can handle execution of multiple web services. What would make this better in terms of usability for users of the system is when a featureName is evaluated to off that we can output to the console or report or both

Step Name (Execute Web Service) At Line Number (Step Line Number) Didn't Execute On This Environment (Environment Type) As Feature Name (featureName) is off

especially if the gherkin step would be repeated often in a test script. That would be the case sometimes for us. Hence why having access to the Step object from the glue context would be very handy. As mentioned in my previous post we can use through the plugin system but it's far more effort to make the step object available to the glue context from the plugin api. Hence the PR submitted would be of interest to us.

Their probably will be strong opinions on the usage of cucumber here admittedly. However for us this works adequately as it allows us to use the same test scripts across multiple environment types plus allowing us to test with features turned on/off too.

Yes. The way you use Cucumber doesn't align with the way Cucumber is intended to be used. And the plugin system also is not intended to solve your specific problem. A scenario in a feature file is supposed to represent the test as executed. It wouldn't be able to function as a non-technical artefact that describes the behaviour of a system if the execution of that behaviour is conditional on that system.

Cucumber does provide several mechanisms to deal with different environments such as tags to select scenarios that depend on certain capabilities of the environment being tested, conditional hooks to do things before/after tagged scenarios and the ability to skip scenarios half way through through JUnits Assume and TestNGs SkipException.

So I suspect this is a variation of the XY Problem. There is a reason that skipping steps makes more sense to you then skipping entire scenarios. Are your scenarios rather long, to the point that maintaining the different variations would be cumbersome? Do they contain lots of detail such as HTTP status codes and verbs, json payloads and responses, urls, ect? If so, you may not actually be describing the behaviour of the system. If this is the case, and you are not interested in changing this because it suits your needs, you may be better of by calling your step definitions from a unit test. You don't need Cucumber as a DSL here because your scenarios are presumably not fit for human consumption.

Well you asked me about the usage I intended and I basically articulated it to you. As I said the usage would likely trigger strong opinions but been honest, it wasn't really something I wanted to open a thread about because the best practices that you are articulating (which I do try to follow most of the time) are sometimes not possible given variant constraints associated with a project which are not visible to folks externally. Regarding tags yes we do use them in our tests to decide what tests to execute for example when running on CI infrastructure we would use tags to run a subset of tests or the whole test inventory.

Regarding our test scenarios we do generally try to keep them compact. We have variant process in place so to reduce technical low-level noise been articulated in the feature file because as you say they don't become usable for human consumption if there is too much detail in them. For example the sample glue function I supplied could has a resource file equivalent which would contain low-level detail like http codes, verbs, json payloads, responses and urls. In addition the sample glue I sent was not exactly what is within our project just a very simple illustration of how I intend to use it which I understand you assert it of not best practice which is fair enough.

The reality is though most people don't use cucumber to the letter of the law it's intended as noted by one of the contributors of cucumber

https://github.com/cucumber/cucumber/issues/1004#issuecomment-631438511

and maybe this is the case here. In any case all of above is a bit of a divergent topic from the original request namely accessing the step information from the glue code. I commented on this because I intend to use the API if the PR was merged which it would seem other users on the thread would benefit from too.

You expressed your support for a feature.

Every feature comes at a cost in terms of development and maintenance. Cucumber is a tool to support BDD. Features that don't contribute to that goal are a burden.

So while being useful is a nessesary property of a feature, it is not sufficient. A feature should ultimately support BDD.

Hence the interest in your use case. It is what determines the weight of your interest.

From what you've described so far I don't see a compelling usecase. Optional steps are not a pattern we are interested in supporting. Additionally it makes no sense for an open source project bear the burden of a professionals need to be pragmatic.

Just to clarify the PR that is open is regarding making the step object available to the glue code. It doesn't introduce optional steps fyi (that was my use case which is already in use, which I know you don't like, we don't need to continue this aspect of the conversation further). Other users expressed an interest as well as myself in having it and ultimately you did mention that you would accept a PR for this

https://github.com/cucumber/cucumber-jvm/issues/1805#issuecomment-626168493

Hence me having an interest in this. However, if this isn't gonna be merged then I'll just have to find a workaround so. Thank you.

@aslakhellesoy

I'm still not convinced why we should add this given that the plugin API exists.

Ran into a pretty reasonable use case.

When testing against a system with multiple components Cucumber tests may cause errors in unexpected parts of the system. Usually because of incomplete assumptions either by the authors of the test or the authors of the system.

Tracing these errors back to an individual test case can be pretty difficult. However using open tracing it is possible to attach scenario names as baggage to http request. This scenario name is then propagated through the entire system including the logging. This makes it possible to quickly find the offending scenarios quickly. Because the HTTP client is a dependency injected into the glue code it makes sense to set the tracing baggage using the before/after hooks.

However a scenario typically involves about 4-6 steps, each consisting of several http request. When a single step fails, this usually impacts multiple scenarios. Including the step with the tracing baggage would make it easier to pin point the failures to a single step.