Tests are sometimes not being re-compiled, leading to incorrect passes or running when they shouldn't compile

[michboon]

What happened?

Using the Gradle IntelliJ plugin is causing unexpected caching behavior leading to tests passing when they shouldn't (or running when they shouldn't compile).

When a test is broken due to a change outside of test code, the test class isn't being re-compiled. This issue seems to be caused by the Gradle IntelliJ plugin because it works fine without.

We believe that this might be caused by some dependency declaration issue that is causing over-caching.

Relevant log output or stack trace

Steps to reproduce

Take a minimal project with Foo & FooTest.

./gradlew test passes as expected.

Now comment out fun foo() and re-run ./gradlew test. It will fail at task compileTestKotlin as expected.

Now add the Gradle IntelliJ plugin into the mix by uncommenting the relevant parts in build.gradle.kts below (also resetting fun foo()). The test will pass on the first run as expected, but when commenting fun foo() and re-running, the compileTestKotlin task will be cached and the test will be run.

In this repro example, the running test will still fail (java.lang.NoSuchMethodError at FooTest.kt:10) but it shouldn't even get to this stage. In our main project, we have also observed even more caching that led to the test "passing".

---

Foo.kt

package org.example

object Foo {

  fun foo(): Boolean {
    return true
  }
}

FooTest.kt

package org.example

import junit.framework.TestCase.assertTrue
import org.junit.Test

class FooTest {

  @Test
  fun exampleTest() {
    assertTrue(Foo.foo())
  }
}

build.gradle.kts

//import org.jetbrains.intellij.platform.gradle.TestFrameworkType

plugins {
    id("java")
    id("org.jetbrains.kotlin.jvm") version "1.9.24"
//    id("org.jetbrains.intellij.platform") version "2.2.1"
    kotlin("plugin.serialization") version "1.9.24"
}

group = "org.example"
version = "1.0-SNAPSHOT"

repositories {
    mavenCentral()
//    intellijPlatform {
//        defaultRepositories()
//    }
}

dependencies {
//    intellijPlatform {
//        intellijIdeaCommunity("2024.3.2.1")
//
//        bundledPlugin("com.intellij.java")
//
//        pluginVerifier()
//        zipSigner()
//
//        testFramework(TestFrameworkType.Platform)
//    }

    testImplementation("junit:junit:4.13.2")
    // Add missing test dependency. See: https://youtrack.jetbrains.com/issue/IJPL-157292
    testImplementation("org.opentest4j:opentest4j:1.3.0")
    implementation(kotlin("stdlib-jdk8"))
}

tasks {
    // Set the JVM compatibility versions
    withType<JavaCompile> {
        sourceCompatibility = "17"
        targetCompatibility = "17"
    }
    withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile> {
        kotlinOptions.jvmTarget = "17"
    }
}

kotlin {
    jvmToolchain(17)
}

gradle.properties

# Opt-out flag for bundling Kotlin standard library -> https://jb.gg/intellij-platform-kotlin-stdlib
kotlin.stdlib.default.dependency=false
# Enable Gradle Configuration Cache -> https://docs.gradle.org/current/userguide/configuration_cache.html
org.gradle.configuration-cache=true
# Enable Gradle Build Cache -> https://docs.gradle.org/current/userguide/build_cache.html
org.gradle.caching=true

Gradle IntelliJ Plugin version

2.2.1

Gradle version

8.10

Operating System

Linux

Link to build, i.e. failing GitHub Action job

No response