Cucumber4: Obtain Cucumber 3 scenario properties in hooks.rb

[orien]

Describe the bug

In version 3 it is possible to obtain the name of the current feature from Before and After hooks.

Before do |scenario|
  feature_name = scenario.feature.name
  #…
end

In version 4 this seems impossible as the feature method has been removed and I can see no replacement.

Before do |scenario|
  feature_name = scenario.feature.name
  # -> undefined method `feature' for #<Cucumber::RunningTestCase::TestCase:0x00007f9b307ae180> (NoMethodError)
end

To Reproduce Steps to reproduce the behavior:

1. Install version 4
2. Add Before hook obtaining feature name
3. Run cucumber
4. See error

Expected behavior

Be able to obtain the running feature name from Before and After hooks.

Context & Motivation

Providing a replacement for this would ease the migration path to version 4.0.0 as integrations rely on this data.

For example, the VCR gem uses the scenario and feature names to determine which cassette to load.

[luke-hill]

Do other methods on the scenario object work. As it seems to be much different.

Examples are scenario.source_tag_names scenario.name e.t.c.?

[orien]

I see the following methods:

Cucumber 3

Before do |s|
  s.exception # => #<NilClass>
  s.feature # => #<Cucumber::Core::Ast::Feature>
  s.keyword # => #<String>
  s.language # => #<Gherkin::Dialect>
  s.location # => #<struct Cucumber::Core::Ast::Location::Precise
  s.name # => #<String>
  s.source # => [#<Cucumber::Core::Ast::Feature>, #<Cucumber::Core::Ast::Scenario>]
  s.source_tag_names # => [#<String>]
  s.status # => #<Symbol>
  s.step_count # => #<Integer>
  s.tags # => [#<Cucumber::Core::Ast::Tag>]
  s.test_steps # => [#<Cucumber::Core::Test::Step>, #<Cucumber::Core::Test::Step>]
end

Cucumber 4

Before do |s|
  s.exception # => #<NilClass>
  s.hash # => #<Integer>
  s.id # => #<String>
  s.language # => #<String>
  s.location # => #<struct Cucumber::Core::Test::Location::Precise
  s.name # => #<String>
  s.source_tag_names # => [#<String>]
  s.status # => #<Symbol>
  s.step_count # => #<Integer>
  s.tags # =>[#<Cucumber::Core::Test::Tag>]
  s.test_steps # => [#<Cucumber::Core::Test::Step>, #<Cucumber::Core::Test::HookStep>]
end

[luke-hill]

Very comprehensive answer - thankyou. I'd definitely expect we could keep the old methods, but I've not been involved much with it recently.

[JoeSSS]

Are there any workarounds to get feature name inside the scenario? We really want to update to new Cucumber version but feature name is essential for our tracking and flakiness monitoring on test results BE.

[thedeeno]

If anyone wants a work-around for VCR while this is being considered I hacked up the following monkey-patch:

# HACK: this method was available in cucumber 3.1 and VCR relies on it to
# generate cassette names.
module Cucumber
  module RunningTestCase
    class TestCase < SimpleDelegator

      def feature
        string = File.read(location.file)
        document = ::Gherkin::Parser.new.parse(string)
        document[:feature]
      end

    end
  end
end

There's probably a better way to do this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[atyndall]

I believe this is still an issue?

[luke-hill]

It is yes, but it is distinctly non-trivial and would require an agreement from the technical team on "what" we should do and whether we standardise across the flavours. I'll pop a slow-burner on it.

[Castone22]

I was doing some quick digging into the JSON formatter because SOMEHOW it still gets the feature name, seems like the expected way to handle it now is to use an AstLookup... https://github.com/cucumber/cucumber-ruby/blob/e318dfcce9083c9dbe9f36c29bff2ce6f7759394/lib/cucumber/formatter/json.rb#L26

which gets referenced here https://github.com/cucumber/cucumber-ruby/blob/e318dfcce9083c9dbe9f36c29bff2ce6f7759394/lib/cucumber/formatter/json.rb#L38

The builder uses this to create a feature hash which can be cross referenced with the test location to get its feature name. https://github.com/cucumber/cucumber-ruby/blob/e318dfcce9083c9dbe9f36c29bff2ce6f7759394/lib/cucumber/formatter/json.rb#L240-L247

Should be able to intercept the config in an AfterConfig hook and save it for use i'd think.

[luke-hill]

@Castone22 That is not using the new message protocol. So it's not a like for like comparison. The legacy JSON formatter is being removed (It was slated for v5 removal, I believe this is now pushed back to v6)

[Castone22]

I suppose that explains why the ast from the ast lookup consists of protofbufs, do you happen to know retrieving those is also slated to be removed?

[rickpastoor]

It appears that the patch by @thedeeno no longer works with cucumber 7.0.0. Investigating further, but if anyone already found the solution I'd love to hear it :)

[luke-hill]

The internal structure of cucumber messages has basically been rewritten from scratch in cucumber 7. We now use a JSON schema instead of protobufs. Consequently, it's likely a lot of the internal structure has changed.

[rickpastoor]

This patch works for me:

module Cucumber
  module RunningTestCase
    class TestCase < SimpleDelegator

      def feature
        string = File.read(location.file)
        document = ::Gherkin::Parser.new.parse(string)
        document.feature
      end

    end
  end
end

[aurelien-reeves]

I put both "Bug" and "Enhancement" labels because historically, it is a regression when cucumber v4 has been released, but regarding all the internal rework which led to the actual v7, internally it would actually be an enhancement :sweat_smile:

[aurelien-reeves]

After our community meeting, it appears that this was actually not a regression, but an expected result from passing a pickle to the hook instead of the AST.

Actually a fix to that issue would be to rename the parameter from scenario to test_case. That way it would reflect better the intent behind it.

To the user who would need access to the data of the feature - like its name - in a hook, why would you do so? What are your use-cases? Your needs?

[rickpastoor]

Thanks @aurelien-reeves! The feature name is used by VCR to name the cassettes of auto-recorded interactions with third parties. If there's a better way to do this I'm more than happy to change that, but I kind of like the VCR folders this way.

[luke-hill]

You can still access the scenario name, the scenario location (Which includes a fully qualified filepath), and things like tags.

The switchover in mentality (I was against it at first too), is that the pickles execute the test cases, i.e. they should be thought of as the top level wrapper (or a scenario if thats easier). The feature isn't a realistic wrapper at this point.

I mentioned VCR were one of the larger consumers of this, I think maybe using a scenario name to qualify your VCR cassettes might help.

[Castone22]

Another use case is having a custom formatter that logs metadata such as relative feature location, id, name and scenario gherkin plus the pass and fail state to with error output and such to an elk instance. (Something we're planning to open source in the near future.)

I'd expect other report tooling that hooks into things like zephyr scale or x-ray would benefit from a way to access this information. We're often more interested in the feature ast data in these situations though. I don't necessarily care about the form the data takes, Json is completely acceptable. We used to be able to resolve this through scenario.feature.source

[mattwynne]

@Castone22 the formatter / events API is a better fit for that, though it's probably not very well documented at the moment. There are examples in Cucumber's own codebase were we match together the data from the test case result and Gherkin AST events to give the kind of info you're looking for, e.g. https://github.com/cucumber/cucumber-ruby/blob/97a03c0bd8199d5dcd3fc8de5993d01306dd4a1f/lib/cucumber/formatter/pretty.rb

It would be possible to build a richer domain model from these events to surface into a user API, but it's not something we have at the moment.

[Castone22]

Yes those is what I ultimately ended up using. A lack of a direct link from a test case to it's ast source was a little awkward at first though using an ast lookup was the pattern I was referring to, not the protobuf that resulted from using it back in 5. I was waiting for confirmation this was the intended route before I continued implementing against it though.

[mattwynne]

@brasmusson really clarified it for me today at the Zoom community meeting, that the hooks are part of an execution model, and we should keep the info exposed in them lean because, for example, we might have test cases that were generated from sources other than Gherkin documents, or test cases running in parallel.

On the other hand, the events API used by the formatters is the higher-level place where you can observe everything going on in the Cucumber run, including AST nodes and execution results.

I think we're probably missing an abstraction layer that makes it easier to work with the events API for common use cases, so please keep talking to us and see if there are patterns in what you do with it that could be pushed upstream.

[aurelien-reeves]

Thanks for all those explanations everyone!

What is that VCR project you are talking about?

[luke-hill]

@aurelien-reeves https://github.com/vcr/vcr it's quite a large gem used in lots of projects (Like rails testing for example).

[aurelien-reeves]

Actually we still have to rename the paramter from "scenario" to "test_case" to reduce the confusion

[aurelien-reeves]

I took a look in order to rename the parameter from "scenario" into "test_case" for hooks. I was not able to find in the code any relevant place.

@luke-hill do you know if it is only matter of documentation? Or does the code need some update too?

Reminder: in order to be more clear than the hook parameter represent a pickle and not a scenario, we wanted to rename Before do |scenario| into Before do |test_case|

[luke-hill]

in code it's kinda irrelevant as it's just a placeholder name.

Just in ruby these names invoke meaning. So if the param is called scenario it would be reasonable to assume you could get scenario based properties from them.

This is purely a doc issue now.

[aurelien-reeves]

in code it's kinda irrelevant as it's just a placeholder name.

Just in ruby these names invoke meaning. So if the param is called scenario it would be reasonable to assume you could get scenario based properties from them.

That's the point: I did not find any code in the repo which had a param named 'scenario'. So I guess our own codebase is ok

This is purely a doc issue now.

👍

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two months if no further activity occurs.

[RamadhanaRey]

Before do |scenario|
  if scenario.respond_to?(:scenario_outline) && scenario.scenario_outline 
    message = "Running example: #{scenario.scenario_outline.name} - #{scenario.name}"
  else
    message = "Running scenario: #{scenario.name}"
  end
end

got this error when run the cucumber scenario_outline undefined method `scenario_outline' for #<Cucumber::Core::Test::Case:

can someone help me to solve this?