Enhancement: enable API compatibility on consumer side

[arnaud-deprez]

Hi,

Currently spring-cloud-contract supports this test matrix:

	Consumer Head	Consumer Prod
Producer Head	Ok	Ok (ensure provider is backwards compatible)
Producer Prod	~~Not Ok~~ (ensure consumer is backwards compatible)	Already tested (assuming)

1. Consumer Prod against Producer Head: It is easily achievable by configuring the Spring Cloud Contract Verifier via spring-cloud-contract-maven-plugin or spring-cloud-contract-gradle-plugin like in apiCompatibility profile in maven or in apiCompatibility gradle task. The idea of this task is to re-run the Spring Cloud Contract Verifier tests but with contracts from production so we can check if the consumers running in production will be compatible with the new provider.

2. Producer Prod against Consumer Head: It is not easy to customise or override the behaviour of @AutoConfigureStubRunner to retrieve stubs from other version so we can test the consumer backward compatibility before deploying the new consumer to the target environment. So the intent of this scenario is not to test the consumer backward compatibility at build time but before deployment to prevent the consumer to use unsupported service interactions. It should be possible to do such operation by playing with the combination of StubRunnerRule JUnit rule and some maven or gradle configuration, but then it is linked to the JUnit way. I don't know yet how to do it in Spock for instance. Moreover, the JUnit rule does not play well with @SpringBootTest initializing phase so the @AutoConfigureMessageVerifier does not work here and you have to define your own MessageVerifier.

I was thinking about using a maven and gradle plugin for such use case so it is also easier to manage it from a CI/CD pipeline.

Although, I'm not sure how to do it as I have to learn the design of spring-cloud-contract. I'm just sharing a desired feature :)

WDYT ?

[OlgaMaciaszek]

@arnaud-deprez Thanks for submitting this issue. With @AutoConfigureStubRunner, you can pass ids of the artifact (prod version or dev version); you can also configure a maven profile, to pass stub ids as system property (different ids set for testing against prod version). Would this help? If not, please provide an example so that we can understand the issue better.

[arnaud-deprez]

Hi @OlgaMaciaszek, thanks for your reply. This would help if it was possible but it does not seem to be. To be clear, I'm using spring-boot 2.0.4.RELEASE and spring-cloud Finchley.SR1 I've tried all these options:

@AutoConfigureStubRunner(stubsMode = StubRunnerProperties.StubsMode.LOCAL, ids = "com.powple.poc:beer-api-producer:${stubs.beer-api-producer.version +}:stubs")
@AutoConfigureStubRunner(stubsMode = StubRunnerProperties.StubsMode.LOCAL, ids = "com.powple.poc:beer-api-producer:#{stubs.beer-api-producer.version}:stubs")

For each, I got :

...
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.powple.poc:beer-api-producer:jar:stubs:${stubs.beer-api-producer.version}

So it does not resolve the system property. Then I tried:

@AutoConfigureStubRunner(stubsMode = StubRunnerProperties.StubsMode.LOCAL, ids = "com.powple.poc:beer-api-producer:#{systemProperty['stubs.beer-api-producer.version'] ?: '+'}:stubs")

And there I got

java.lang.IllegalStateException: Failed to load ApplicationContext

	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:125)
	at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:108)
	at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
	at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
	at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:246)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'stubFlowRegistrar' defined in class path resource [org/springframework/cloud/contract/stubrunner/messaging/stream/StubRunnerStreamConfiguration.class]: Unsatisfied dependency expressed through method 'stubFlowRegistrar' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'batchStubRunner' defined in class path resource [org/springframework/cloud/contract/stubrunner/spring/StubRunnerConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.contract.stubrunner.BatchStubRunner]: Factory method 'batchStubRunner' threw exception; nested exception is java.util.NoSuchElementException
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:732)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:474)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1256)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1105)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:503)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:139)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
	... 24 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'batchStubRunner' defined in class path resource [org/springframework/cloud/contract/stubrunner/spring/StubRunnerConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.contract.stubrunner.BatchStubRunner]: Factory method 'batchStubRunner' threw exception; nested exception is java.util.NoSuchElementException
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:590)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1256)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1105)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:503)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1065)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:818)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:724)
	... 42 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.contract.stubrunner.BatchStubRunner]: Factory method 'batchStubRunner' threw exception; nested exception is java.util.NoSuchElementException
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:582)
	... 55 more
Caused by: java.util.NoSuchElementException
	at java.util.LinkedList.removeFirst(LinkedList.java:270)
	at java.util.LinkedList.pop(LinkedList.java:801)
	at org.springframework.cloud.contract.stubrunner.StubRunnerOptionsBuilder.stubsToList(StubRunnerOptionsBuilder.java:189)
	at org.springframework.cloud.contract.stubrunner.StubRunnerOptionsBuilder.withStubs(StubRunnerOptionsBuilder.java:63)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration.builder(StubRunnerConfiguration.java:91)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration.batchStubRunner(StubRunnerConfiguration.java:70)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration$$EnhancerBySpringCGLIB$$ecf6c5d3.CGLIB$batchStubRunner$0(<generated>)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration$$EnhancerBySpringCGLIB$$ecf6c5d3$$FastClassBySpringCGLIB$$d59f7b9d.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
	at org.springframework.cloud.contract.stubrunner.spring.StubRunnerConfiguration$$EnhancerBySpringCGLIB$$ecf6c5d3.batchStubRunner(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 56 more

If you have another way, I can try.

[marcingrzejszczak]

Is this still a problem?

[arnaud-deprez]

I think it's still a nice to have unless it has already been resolved. Although I didn't have the time and this can be mitigated with some issue management tool.

The idea behind is to validate that a new api consumer use a valid contract with the existing provider.

One concrete use case I see is when a consumer is using a new API on which the producer has agreed at some point (so latest contract tests pass) but has still not implemented it (in production for example).

[OlgaMaciaszek]

@arnaud-deprez could you verify if the problem also exists in Greenwich + 2.1.x?

[arnaud-deprez]

I'm on vacations. Will be back on Tue 20.

[arnaud-deprez]

Hi,

As you requested, I've tested with latest version and it still does not work.

So this is working with junit rule where I can specify the stub version to use via system property or environment variable like this: https://github.com/arnaud-deprez/spring-cloud-contract-howto/blob/feature/consumer-check/beer-api-consumer/src/test/java/com/powple/poc/BeerControllerWithJUnitTest.java#L40-L43

However this does not work: https://github.com/arnaud-deprez/spring-cloud-contract-howto/blob/feature/consumer-check/beer-api-consumer/src/test/java/com/powple/poc/BeerControllerYamlTest.java#L34

It could be solved if JUnit rule or extension (Junit 5) works with contract messaging as well but it's not (as per doc and I also double check it). But this solution would be stick to a test framework (JUnit in this case).

That's why I think it's better if the ids in @AutoConfigureStubRunner could be evaluated as spring expression to resolve against jvm properties and environment variables.

[marcingrzejszczak]

I agree. I think i started some work on this but then i had to migrate to sth else.