Gradle projects don't run out-of-the-box with JUnit5 mode

[jamietanna]

Describe the bug

When running with only JUnit5 on the classpath, the tests get generated successfully, but they do not execute.

This is due to the useJUnitPlatform configuration for the contractTest in Gradle not being called. We should default this when we are using TestFramework.JUNIT5.

Sample

plugins {
  id "org.springframework.cloud.contract" version '3.1.0'
}

dependencies {
  testImplementation 'org.springframework.cloud:spring-cloud-starter-contract-verifier:3.1.0'
}

contracts {
  testFramework = TestFramework.JUNIT5
}

// TODO: this needs to be uncommented to make it work
// contractTest {
//   useJUnitPlatform()
// }

[jamietanna]

Initially it looks like the following diff may be what we need:

diff --git spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/java/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.java spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/java/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.java
index 8c52c12d25..f79fdf28fe 100644
--- spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/java/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.java
+++ spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/java/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.java
@@ -122,6 +122,7 @@ public class SpringCloudContractVerifierGradlePlugin implements Plugin<Project>
 				else {
 					contractTestSourceSet.getJava().srcDirs(generatedTestSourcesDir);
 				}
+				project.getTasks().withType(Test.class).getByName("contractTest").useJUnitPlatform();
 			}
 		});
 	}

I'll look at setting up a new testing example in the Gradle plugin's functionalTest folder to verify this, though

[marcingrzejszczak]

@shanman190 WDYT?

[shanman190]

So the suggestion is close, but there are some potential gotchas which would at minimum change it's location.

Gradle JUnit 5 support does require useJUnitPlatform() to be configured. Now there are some issues with doing this though and where you enable this.

Early versions of Gradle 7, introduced a feature that throws an error when you change the test runtime engine after options are configured (some options are specific to the test engine). This has since been reverted in Gradle 7.3.1 -- instead logging a warning, rather than throwing an error as there was a lot of end-user pain -- it would be in our best interest to configure this as soon as possible, so as to not end up with issues being created as a result of this configuration and end-user option configurations.

Since the plugin itself creates the contractTest task, my suggestion would be to go ahead and configure JUnit Platform alongside it during it's creation as the default framework is TestFramework.JUNIT5.

https://github.com/spring-cloud/spring-cloud-contract/blob/b14ba2b5b94d5d903687f236b52abad52a2c1c82/spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/java/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.java#L185

This also probably warrants a documentation mention that TestFramework.SPOCK needs to have JUnit Vintage Engine included (I can't recall there being one right now) in order to continue functioning. Given the impact of Spock tests outright failing without JUnit Vintage Engine being included, I'm assuming that this particular change will need to roll up into the next minor release, but @marcingrzejszczak will need to make that call.

In the interim, the workaround would be to explicitly enable useJunitPlatform() for the contractTest.

[marcingrzejszczak]

Since the plugin itself creates the contractTest task, my suggestion would be to go ahead and configure JUnit Platform alongside it during it's creation as the default framework is TestFramework.JUNIT5.

:+1: we can add it to 3.1.x version.

This also probably warrants a documentation mention that TestFramework.SPOCK needs to have JUnit Vintage Engine included (I can't recall there being one right now) in order to continue functioning. Given the impact of Spock tests outright failing without JUnit Vintage Engine being included, I'm assuming that this particular change will need to roll up into the next minor release, but @marcingrzejszczak will need to make that call.

:+1: we should add the docs about this for 3.0.x branch.

WDYT?

[shanman190]

Yeah, I think that looks good @marcingrzejszczak. Especially with -- I think it was Boot 2.4 -- no longer including the vintage engine.

[marcingrzejszczak]

OK @jamietanna are you willing to apply the beforementioned changes?

[jamietanna]

Looking at adding a functionalTest for this, and I see that the existing tests are @Ignored as part of #1378

Is there an alternative that's recommended, or i.e. running them only locally as part of testing? Or are we happy me making the changes without tests (:sweat_smile:)?

[shanman190]

@jamietanna, I'm particularly on a Windows machine and was able to temporarily drop the @Ignored to test. The other part of testing was pulling down the https://github.com/spring-cloud-samples/spring-cloud-contract-samples. It appears that main tracks 3.1.x, so building the Gradle plugin using Maven was how @marcingrzejszczak showed me to get it going, then running the ./scripts/runGradleBuilds.sh in the samples will get you going with testing the various samples.

[jamietanna]

That's interesting, I've found that making the following change:

diff --git spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/test/groovy/org/springframework/cloud/contract/verifier/plugin/BootSimpleSpec.groovy spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/test/groovy/org/springframework/cloud/contract/verifier/plugin/BootSimpleSpec.groovy
index de88cbf296..7ddd6b32a0 100755
--- spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/test/groovy/org/springframework/cloud/contract/verifier/plugin/BootSimpleSpec.groovy
+++ spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/test/groovy/org/springframework/cloud/contract/verifier/plugin/BootSimpleSpec.groovy
@@ -21,7 +21,6 @@ import org.gradle.testkit.runner.TaskOutcome
 import spock.lang.Ignore
 import spock.lang.Stepwise
 
-@Ignore
 @Stepwise
 class BootSimpleSpec extends ContractVerifierIntegrationSpec {
 
@@ -29,6 +28,7 @@ class BootSimpleSpec extends ContractVerifierIntegrationSpec {
                setupForProject("functionalTest/bootSimple")
                runTasksSuccessfully('clean')
                //delete accidental output when previously importing SimpleBoot into Idea to tweak it
+               assert(false)
        }
 
        def "should pass basic flow for Spock"() {

Then running ./gradlew clean test doesn't fail :thinking:

[shanman190]

@jamietanna, yeah I found the same thing last time. I think there were issues when it executed in CI which is why it ended up getting disabled.

@marcingrzejszczak, do you recall what the issue was there?

[marcingrzejszczak]

Yeah there were some CI issues and we have a lot of different combinations tested in the standalone samples + the spring cloud contract samples repo so I guess this is why those got disabled. We can remove them actually.

[marcingrzejszczak]

@shanman190 anything we should do about this after a year? :grimacing:

[shanman190]

@marcingrzejszczak, so this is a little funny. Since I added the jvm-test-suite support and noticed that the default was JUNIT5 mode, I went ahead and turned on useJUnitPlatform() during that PR. You can see that here: https://github.com/spring-cloud/spring-cloud-contract/blob/c04b110fe2a4f9e526f397fcd2b3f67e7afa79df/spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/java/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.java#L137

I think with that said, that means that this got resolved in the process. 😆

To handle the issues from my previous comment:

• We now only support Gradle 7.4 or newer, so we don't have to worry about the error being thrown from switching options, if that arises.
• With spock new versions now support junit 5 native mode or old versions can include the vintage engine.

[marcingrzejszczak]

Fixed via https://github.com/spring-cloud/spring-cloud-contract/issues/1863