jenkins-spock does not mock 'sh' when passing additional arguments

[illegalnumbers]

Expected Behavior

passing getPipelineMock("sh")([returnStdout:true], cmd) for the 'sh' step of workflow-basic-steps properly mocks and returns a mocked variable.

Actual Behavior

When passing additional arguments (returnStdout, returnStatus) nothing is mocked properly.

Steps to Reproduce

Mock a call with a extra variable and see that nothing is returned appropriately.

[illegalnumbers]

Also doesn't seem to work if you set your scripts or other hash variables to wildcard _.

[awittha]

Could you elaborate a bit on your test setup? E.g., how your code-under-test is calling sh(...) and what invocation(s) you're trying to capture?

As far as I can tell, [returnStdout:true], cmd is not a set of parameters that valid pipeline code could ever generate for sh(...) - not code that Jenkins would ever be able to dispatch.

The ShellStep has one @DataBoundConstructor that takes a String (source) - this is what is bound to sh( "echo 'hello world'" ) and similar.

Any other invocation must use named parameters for all parameters, as a single Groovy Map, e.g.

sh( returnStdout: true, script: "echo 'hello world'" )

This will invoke all of the set<namedParameterName>(...) setters to configure the step, and you'd identify that mock with

getPipelineMock( "sh" )( [returnStdout: true, script: "echo 'hello world'"] )

Mocking / Stubbing / Asserting things about pipeline steps invoked with named parameters can be tricky, though, as Spock will be checking "equality" among the input map and the map in your test suite.

If the underlying implementations are different and order matters - or if you've got complex objects in the input map that don't override the equality operator to check semantic equality, Spock may conclude that the invocations' arguments didn't match your assertion even though to you, the human, things look "the same".

I'll try to put together a working example of how to mock a returnStdout'd sh step...

[awittha]

This Jenkinsfile:

https://github.com/homeaway/jenkins-spock/blob/issue-50/examples/whole-pipeline/Jenkinsfile#L1-L4

And this test:

https://github.com/homeaway/jenkins-spock/blob/issue-50/examples/whole-pipeline/src/test/groovy/JenkinsfileSpec.groovy#L12-L16

demonstrate how to assert & stub a call to the sh(...) pipeline step that uses named parameters.

It is a characteristic of Spock itself that you cannot stub in setup: and also assert in then: for the same mock object - so in this case, to assert a # of invocations and also stub them, the stubbing must happen in then:.

To just stub, you could move it up to setup:

setup:
	getPipelineMock("sh")( [returnStdout: true, script: "echo 'hello world'" ] ) >> "hello world"
when:
	Jenkinsfile.run()
then:
	...

To just assert, simply remove the >> stubbed response from the line in then:

[awittha]

sh( returnStdout: true, "echo 'hello world'") does not evaluate on an actual Jenkins server:

Jenkinsfile:

node {
    sh( returnStdout: true, "echo 'hello world'" )
}

java.lang.IllegalArgumentException: Expected named arguments but got [{returnStdout=true}, echo 'hello world']
	at org.jenkinsci.plugins.workflow.cps.DSL.parseArgs(DSL.java:588)
	at org.jenkinsci.plugins.workflow.cps.DSL.parseArgs(DSL.java:526)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:220)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
...

[illegalnumbers]

https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script is how the stdout and other options is configured

[illegalnumbers]

So it was my mistake in the specific library, it should be from the durable task.

[illegalnumbers]

Oh, ah, typo there. I missed the script call specifically. It should be getPipelineMock("sh")([returnStdout:true, script: cmd])

[illegalnumbers]

This is one of the console messages I get:

12:58:17.285 [Test worker] DEBUG com.homeaway.devtools.jenkins.testing.JenkinsPipelineSpecification - TEST FIXTURE intercepted & redirected a call:
	Test         : class com.homeaway.devtools.jenkins.testing.JenkinsPipelineSpecification
	Note         : pipeline step 
	Via          : [email protected]
	    (types)  : thing.methodMissing
	Invocation   : [email protected]([[script:
                           ... my script ....
                        , returnStdout:true]])
	    (types)  : thing.sh(Object[])
	Forwarded To : Mock for type 'Closure' named 'getPipelineMock("sh")'.call([script:
                          ...my script...
                        , returnStdout:true])
	    (types)  : Closure$$EnhancerByCGLIB$$4117dc53.call(LinkedHashMap)

[illegalnumbers]

I am running my setup for my mock in the setup function as well.

[awittha]

Having updated your mock/stub to use just one parameter: a map with key/value pairs for each named parameter of sh, are you able to get the code-under-test to connect with the mocks/stubs?

[illegalnumbers]

I haven't been able to no, I have tried something like the following:

 getPipelineMock("sh")([script: _, returnStdout:_]) >> { args ->
            System.out.println(args)
            return 'thing'
        }

as well as

 getPipelineMock("sh")([script: scriptInVar, returnStdout: true]) >> { args ->
            System.out.println(args)
            return 'thing'
        }

My situation is that I have many steps that call sh and I need some of them to have mocked return values to be consumed by others. The steps that I want to return don't seem to be getting called from the mock and are not returning anything.

[awittha]

Ahh, I see.

Specifically regarding this:

Also doesn't seem to work if you set your scripts or other hash variables to wildcard _.

That's where

Mocking / Stubbing / Asserting things about pipeline steps invoked with named parameters can be tricky, though

comes in.

The _ wildcard is a Spock idiom. I do not believe spock applies this recursively, so if you set [returnStdout: true, script: _], Spock sees a Map and doesn't process the wildcard. Unless your invocation is sh( returnStdout: true, script: _ ), it won't match.

To handle this kind of wildcard-ness, you need to capture the arguments of your assertions, and then assert things about the argument values inside them (!). Check out this document: https://mrhaki.blogspot.com/2013/05/spocklight-writing-assertions-for.html

Off the top of my head, I don't know how you reconcile argument-capture-for-validating-params-to-match-an-assertion with stub-a-response.

I've got to attend to other tasks ATM so I won't be able to continue to engage in real-time. I've put this on my to-do list and will check back when I can to see if you've made progress, and try to offer a working example if not.

If you could

distill your use-case into a few working-but-not-sensitive examples (i.e. a *Spec.groovy and a source-file-to-test that work together) that I can run and see the actual, undesired behavior, that would greatly accelerate my investigation of this.

[illegalnumbers]

Gotcha no worries. I can try and set that up when I get time. I just need to get unblocked from this for now to get moving on my own ticket :) The wildcard definitely doesn't seem to work but also matching on my legit parameters doesn't seem to work either. I'll try and prep examples later.

[stepanataccolade]

@awittha I've run into the same issue. Trying to stub sh return when called like this:

pipenvVersion = sh(script: 'pipenv --version', label: 'Get pipenv version', returnStdout: true).trim()

unless my mocking was done with explicit exact match of parameters used - like this:

getPipelineMock('sh')([script: 'pipenv --version', label: 'Get pipenv version', returnStdout: true]) >> "test"

I was getting a NullPointer exception for the trim() method.

In my case, I would love to use something like (as I have multiple sh calls in the script):

getPipelineMock('sh')([script: _, label: _, returnStdout: true]) >> "test"

or to be honest even something like this:

getPipelineMock('sh')(*_) >> "test"

would be a great start as well

Reading the above, specially this:

Off the top of my head, I don't know how you reconcile argument-capture-for-validating-params-to-match-an-assertion with stub-a-response.

I wonder if you happened to stumble across a solution since then that would allow using the _ for some of the parameters (like in Mockito frameworks where one can use mix of specific values and any() to not to have to provide exact matching literals which makes the tests hard to maintain)

[awittha]

You're, right, this does not work:

getPipelineMock('sh')([script: _, label: _, returnStdout: true]) >> "test"

This is because Spock - the Groovy testing framework on which jenkins-spock is based - only supports the syntactic sugar matchers in arguments to the mock. In this case, there is one argument: it is a Map, and that Map-typed argument's value is not _ - it's actually got a real set of keys and values. That Map will be tested with == against inputs to see if it matches.

[script: _] does not == [script: "pipenv --version"] in Groovy. spock does not deeply recurse looking for possible matchers.

This will work:

getPipelineMock('sh')(*_) >> "test"

But the * is pointless - there is only ever one argument - a Map. spock will not expand *_ into key/value pairs of the map and try to match a k/v pair.

You must do this by hand. Manually, it looks like (probably - not actually tested, but the "magic" you're looking for is the "Argument Constraints" section of the Spock documentation)

getPipelineMock('sh')({
    it instanceof Map &&
    it["script"] == "pipenv --version"
}) >> "test"

etc, etc.

You could maybe write up a helper closure (also not actually tested, but, hopefully you get the idea), like

def mapContains = { expected, actual ->
    return actual?.entrySet().containsAll( expected?.entrySet() )
}

def mapMatches = { expected ->
    return mapContains.curry( expected )
}

Then you could, in your tests, write:

// matches sh(script: "pipenv --version", returnStderr: true)
getPipelineMock("sh")(mapMatches([script: "pipenv --version"])) >> "test"

// matches sh(script: "id", returnStdout: true)
getPipelineMock("sh")(mapMatches([returnStdout: true])) >> "testStdout"

You still will not get to do this:

getPipelineMock("sh")(mapMatches([script: _])) >> "test"

because in this mode, there is no spock-driven argument matching happening; Spock's just feeding the actual arguments into your provided closure.

However, you could expand the logic in mapContains to support _ as an always-match value if expected contained it. You could even make it deeply-recursive!

Maybe some matching utilities like this should be part of jenkins-spock... if you get the suggestion above actually working, perhaps share it!

[stepanataccolade]

Thanks for a quick response, trying out the suggested

getPipelineMock('sh')(_) >> "test"

actually does work, problem was between chair and the keyboard 🙄 as I was getting class cast exception and have not realized my case was in the jenkins var sh was used multiple times and I have used returnStdout somewhere and returnStatus at some spots and then checking with if status>0 which worked in Jenkins (implicit string to int conversion) but not in the tests - obviously (try to convert test string to integer ...)

Unfortunately spock framework did not give the actual .groovy line where the exception occured, but this:

java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.String (java.lang.Integer and java.lang.String are in module java.base of loader 'bootstrap')
        at pythonEnsurePipenvSpec.default container used is python-build(pythonEnsurePipenvSpec.groovy:17)

which did not hit me into the eye where the issue lies.

Thanks for the reply, made me realize my issue.

I'm beginner with spock, so need to educate myself further on the above, will try your suggestions later and definitely will share the findings!

[awittha]

If you're running Spock with Maven, the actual test-runner is the maven-surefire-plugin which will kick off the Spock tests. In that case, you may find a complete stack trace by providing -DtrimStackTrace=false as a CLI flag.

If running with gradle, I don't know.