Karate-Netty: Variables defined in calling feature are not present in the called features.

[fscarlato]

karate-sample-persistent-state-dynamic-scope.zip

Hello! I don't know if is intended or not, but the dynamic scope in MockServer features doesn't work, the test fails in the last line of the second proxy.feature Scenario.

proxy.feature

    * def outer_scope_variable = "hello"
    * call common_outer_scope  ERROR: org.graalvm.polyglot.PolyglotException: ReferenceError: "outer_scope_variable" is not defined

The call to common_outer_scope fails because it can't find the outer_scope_variable defined anywhere. Also the built-in objects, like request\response are not accessible. This is only for karate-netty, in client features everything works as expected:

first-case.feature:

  * def outer_scope_variable = "hello"

  # Here the feature just logs the value of the `outer_scope_variable`, to demostrate
  # that the problem is only in karate-netty feature files
  * call common

I found another bug while investigating this, as you can see in proxy.feature, to the first call to common_request is specified an argument.

Scenario: pathMatches('resource')

    * match request == body_req

    * call common_request ({ test: 1, default_argument: "not default anymore" })
    ...

In the second call btw, (the second Scenario), the argument is not passed, but the called feature continues to see the old value.

Scenario: pathMatches('resource/bad')
    
    * match request == body_req_2

    * call common_request ({ test: 10 })

common-request.feature

    * def default_argument = karate.get('default_argument', "default value")
    * karate.log("Default Argument: ", default_argument)

EveryTimes it prints:

Default Argument: 'not default anymore'

This can endanger behaviours of feature files which relies upon defaults, making them not trustworthy.

[ptrthomas]

tagging as help wanted - we spent a lot of time requesting for feedback on RC versions, and it sounds like this team has used very advanced patterns which need more work.

[ivangsa]

hi, I had a look yesterday and this project is overly complex

the problem seems to be, on a server side feature, when calling another scenario with no arguments, the called scenario can not see the global variables from calling scenario

Attached is a minimun project that shows this error... karate-mocks-server-calling-other-scenario-error.zip

[ivangsa]

hi @fscarlato

this is a workarround it will work in your case, if you access the variable with karate.get('outer_scope_variable')

# common-outer-scope.feature
Scenario: Matching request field

    * karate.log('Outer scope variable in sub-feature called from client feature:', karate.get('outer_scope_variable'))

[fscarlato]

Hi @ivangsa! Yes the local workaround works, but we already fixed this passing the arguments needed from the global scope to the called feature. Like this:

* call common_outer_scope({outer_scope_variable: value})

Our large codebase is already fixed in this way regard to this bug, what do really concerns me is the second bug, the one that makes the state persistent in called features from Scenario to Scenario. A called feature could make decisions and match things generated from other Scenarios, making the tests not trustable anymore. Should I open a new issue for the state persistentness in bug?

Note btw that with the workaround one still can't use the utilities for matching request parts and do things like:

* match $.a == 'myvalue'

Thanks for the following!

[ptrthomas]

@fscarlato @ivangsa just checking if this commit fixes this problem, I don't yet have time to go through the samples: https://github.com/karatelabs/karate/pull/1861#issuecomment-1073254809

[fscarlato]

karate-persistent-scenarios-state.zip No, I'm sorry, but it doesn't fix this problem, I compiled the devel branch, and installed it with mvn install.

I've attacched the same example with the karate version updated to 2.00 and an edit:

When the first scenario hits the /resource path on the proxy feature file, i added * call common_request ({ test: 1 }) before * call common_request ({ test: 1, default_argument: "not default anymore" })

To actually show on the logs that the first time we have the default value.

15:58:24.313 [armeria-common-worker-epoll-2-1] DEBUG com.intuit.karate - scenario matched at line 28: pathMatches('resource') 15:58:24.394 [armeria-common-worker-epoll-2-1] INFO com.intuit.karate - Default Argument: default value 15:58:24.430 [armeria-common-worker-epoll-2-1] DEBUG c.intuit.karate.core.ScenarioEngine - over-writing existing variable 'default_argument' with new value: karate.get('default_argument', "default value") 15:58:24.433 [armeria-common-worker-epoll-2-1] INFO com.intuit.karate - Default Argument: not default anymore

Then in the second test:

Scenario: pathMatches('resource/bad')
    * match request == body_req_2
    * call common_request ({ test: 10 })
    ...

15:52:11.496 [armeria-common-worker-epoll-2-3] DEBUG com.intuit.karate - scenario matched at line 53: pathMatches('resource/bad') 15:52:11.515 [armeria-common-worker-epoll-2-3] DEBUG c.intuit.karate.core.ScenarioEngine - over-writing existing variable 'default_argument' with new value: karate.get('default_argument', "default value") 15:52:11.521 [armeria-common-worker-epoll-2-3] INFO com.intuit.karate - Default Argument: not default anymore

You can notice that it gets printed "not default anymore" instead of "default value"

The second scenario still fails because of the last two lines in the proxy.feature

* def outer_scope_variable = "hello"
* call common_outer_scope ()

Which also shows that in a netty feature file the dynamic scope isn't inherited by the called feature.

[ptrthomas]

@fscarlato thanks. sounds like a fix won't make it into the next major version.

mocks are supposed to be simple :(

[ptrthomas]

this is fixed in develop, the runtime for mocks is a little different from the normal one, but now we detect if a mock is in play and act accordingly

[ptrthomas]

1.3.0 released