Package level ProjectConfig

[tibtof]

For my integration tests, I use a lot of extensions (e.g. TestContainers, Wiremock, Spring), but this setup slows down running individual unit tests a lot. I think it would be useful to have ProjectConfig bound to a single package (including sub-packages) instead of a whole module or to have a more granular PackageConfig. Is there a simple way that I'm missing to already achieve this?

[sksamuel]

There's no current way to do this. I guess we could add a package level project config.

[tibtof]

I would like to work on this, but I need some guidelines. I don't know if a package level config would be flexible enough. Another use case that I can think of is with maven's Surefire plugin for unit tests and Failsafe for integration tests. Would it be better to split the unit and integrations tests at TestEngine level?

[sksamuel]

I wouldn't want to split the tests because person A might want unit/int tests, and person B might want unit/int/smoke tests and so on. It wouldn't be feasible to cover everyone's use cases.

What about package level config wouldn't be flexible enough. If you have an example, I may be able to think of another solution.

[tibtof]

I was thinking that a tag level config might be more flexible and explicit than a package level config.

[sksamuel]

That is a good idea. If we can think of appropriate syntax.

Either a map of configs inside project config, Or an AbstractTagConfig that requires you to define the tags it is active for.

On Tue, 21 Dec 2021 at 16:01, Tiberiu Tofan @.***> wrote:

I was thinking that a tag level config might be more flexible and explicit than a package level config.

— Reply to this email directly, view it on GitHub https://github.com/kotest/kotest/issues/2739#issuecomment-999122181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVSGUMVBRWUJTL2T5I5KTUSD2LZANCNFSM5KI6GA5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

[tonicsoft]

I have also faced this issue in a gradle project.

We considered solving it by splitting the tests into different gradle modules. It seemed like a good solution, I'm interested to here from @tibtof how this is different to his suggestion of splitting at TestEngine level (I'm not familiar with the TestEngine), and also from @sksamuel why separate modules for unit/int/smoke tests wouldn't have worked in whatever scenario you had in mind.

The solution we actually implemented was to lazily acquire expensive project wide resources (docker containers, etc) the first time they are used, rather than eagerly in the beforeProject function. Essentially the implementation is backed by some static global state which is not ideal, but it was a cheap, fast solution that worked.

[tonicsoft]

Another idea which may be worth considering is what do people actually want to achieve here? It sounds like both me and tibtof have the same use case: we want to use project extensions, but only have them activate when they are actually required, i.e. not when running individual unit tests.

What about some kind of API that allowed you to install extensions on individual specs (thus keeping the declaration of the dependency where it is actually used rather than off in some hard to locate ProjectConfig file at the correct level in the package structure, or reached by trying to figure out how a given project has organised their custom tags), but there was some way to execute project wide before/after logic associated with that extension only if at least one test that used it is being run?

Perhaps an option to add support for lazy project wide extensions as a first class citizen rather than complicate the ProjectConfig resolution rules? Especially without clear signals from multiple different users and use cases that this would be useful.

[tonicsoft]

First class support for lazy project wide extensions could amount to configuring the extension the same as one would today, in the ProjectConfig, but declaring a dependency on it within the Spec with an API similar to install:

class Test: DescribeSpec({
    val normalExtension = install(MyExtension())

   val lazyProjectWideExtension = installProjectWide<ProjectExtension>()
})

This also has the potential to bring the benefits of the install vs mount API to project wide extensions, to separate the API of the extension class from the API intended to be exposed to users of the extension.

[sksamuel]

Maybe something like:

val normalExtension = install(MyExtension(), shared = true)

In which case it shares with another spec and the shutdown logic is deferred to the end of the project.

A question would be if you did

val normalExtension = install(MyExtension(1), shared = true)

and

val normalExtension = install(MyExtension(2), shared = true)

Does it just ignore the 2nd one and return the first.

[tonicsoft]

I think you have to pass the same instance in both specs, otherwise you are somehow hiding what is really happening. Maybe this allows you to get rid of shared = true too because the framework can use object reference equality to see if the extension is shared.

How would the framework know which extensions to check for this sharing?

You could have some new listener interfaces that identify this special treatment.

class MyExtension : MountableExtension<Foo, Bar>, BeforeFirstSpec, AfterLastSpec {
    fun beforeFirstSpec(...) {} // effectively a lazy version of beforeProject
    fun afterLastSpec(...) {} // similar to afterProject, but only called if beforeProject was called.
}

Remaining question is where to create the Extension instance and how to inject it into each Spec so they can install it (and thus register a "usage" of the extension, ensuring beforeFirstSpec will be called if needed) and access the materialised value. Can it be done without relying on a global object or val, which seems to come back to bite someone usually?

[sksamuel]

If you restrict to the same instance, then you have to share that instance, unless you mean same input parameters, but then what is "equal'.

I need a use case to drive the design. I think a shared test container is a good one. You want to put a MySQL instance in a bunch of tests. In some tests you want to share, others you want a fresh instance.

I don't like the idea of it doing this magically, so some kind of flag or different method name is best I think.

The beforeFirstSpeccould work but then you put the onus on extension implementations to support it, when it would also work to just control whenbeforeSpec` is called. But maybe the extra delineation is nice as extensions might want to be cleverer if it's shared. Like a reset between specs but not a full restart. The name could be better though.... but the idea is good.

[tibtof]

A typical use case for integration tests is this: start a server on a random available port (using an embedded server or testcontainers) then have multiple tests using the same instance of the server. As a workaround, instead of using ProjectConfig to start the servers, which leads to a long startup time even for a unit test, I use a common superclass (e.g. for wiremock). This is obviously not very extensible.

abstract class IntegrationSpec(body: IntegrationSpec.() -> Unit = {}) : FreeSpec(body as FreeSpec.() -> Unit) {

    companion object {
        private val wireMockConfiguration = WireMockConfiguration()
            .httpsPort(findAvailableTcpPort(8081))

        val wiremock = WireMockServer(wireMockConfiguration)
    }

    override fun beforeSpec(spec: Spec) {
        if (!wiremock.isRunning) {
            wiremock.start()
            System.setProperty("mock.server.port", wiremock.httpsPort().toString())
        }
    }

    override fun afterSpec(spec: Spec) {
        wiremock.resetAll()
    }
}

A tag level config might be difficult to manage: all the tests that will run have to be scanned to find the union of all the tags that are actually used, then apply the tag configs apriori (is that even possible, e.g. when running all the tests from a package in Intellij)?.

A lazy and shared extension seems the better solution because it's less magical and each test could cherry-pick only the extensions that it needs.

[tonicsoft]

I resonate with @tibtof 's use case and have implemented exactly the same base class oriented solution in several old java/junit4 projects. It always causes some minor inconvenience eventually. For what it's worth, here is some code that implements a shared dynamoDb container, just to add some more variety into the examples given already with wiremock and mysql:

class DynaliteContainerExtension : AfterProjectListener {
    val dynamoDbClient get() = dynaliteContainer.client

    companion object {
        private val dynaliteContainerDelegate = lazy {
            val container = DynaliteContainer("quay.io/testcontainers/dynalite:v1.2.1-1")

            container.start()

            container
        }

        private val dynaliteContainer: DynaliteContainer by dynaliteContainerDelegate
    }

    override suspend fun afterProject() {
        if (dynaliteContainerDelegate.isInitialized()) {
            dynaliteContainer.stop()
        }
    }
}

object ProjectConfig : AbstractProjectConfig() {
    private val dynamoExtension = DynaliteContainerExtension()
    val dynamoDbClient get() = dynamoExtension.dynamoDbClient

    override fun extensions() = listOf(dynamoExtension)
}

dynamoDbClient is supposed to emulate the "materialised value" of MountableExtension, accessed statically through the project config object. The idea is that test isolation is maintained by a different Spec level extension that creates a fresh table for each spec, which has drastically less overhead than different docker containers.

Credit to my colleague @zakhenry for this implementation. Given the simplicity of it, is any change to kotest actually needed? It would be nice if there was away to avoid global state and also put the container.start() call somewhere more descriptive like beforeProject. But is it really needed?

[tonicsoft]

You want to put a MySQL instance in a bunch of tests. In some tests you want to share, others you want a fresh instance.

Downside of the above DynaliteContainerExtension is that it does not support the "others you want a fresh instance" requirement. Fulfilling this requirement would indicate the API is really consistent and modular. I would have to create a different extension for that use case that would essentially be a copy and paste of this one which is a bit of a smell.

[tibtof]

@tonicsoft your solution gave me an idea to combine project lifecycle with spec lifecycle. We need AfterProjectListener to be able to register a cleanup stage. The downside is that we need two extensions to separate beforeSpec and afterProject.

import io.kotest.core.config.AbstractProjectConfig
import io.kotest.core.listeners.AfterProjectListener
import io.kotest.core.listeners.BeforeSpecListener
import io.kotest.core.spec.Spec
import io.kotest.core.spec.style.FreeSpec

class FakeEmbeddedServer() {
    fun start() {
        println("Starting fake embedded server...")
    }

    fun stop() {
        println("Stopping fake embedded server...")
    }
}

object EmbeddedContainerProjectExtension : AfterProjectListener {
    val embeddedServer: FakeEmbeddedServer by lazy {
        FakeEmbeddedServer().apply { start() }
    }

    override suspend fun afterProject() {
        embeddedServer.stop()
    }
}

object EmbeddedContainerExtension : BeforeSpecListener {
    override suspend fun beforeSpec(spec: Spec) {
        EmbeddedContainerProjectExtension.embeddedServer
    }
}

object ProjectConfig : AbstractProjectConfig() {
    override fun extensions() = listOf(EmbeddedContainerProjectExtension)
}

class AnUnitSpec : FreeSpec({
    "unit test" {
        println("Running UnitSpec")
    }
})

class FirstIntegrationSpec : FreeSpec({
    extension(EmbeddedContainerExtension)

    "using embedded server" {
        println("Running FirstIntegrationSpec")
    }

})

class SecondIntegrationSpec : FreeSpec({
    extension(EmbeddedContainerExtension)

    "using embedded server" {
        println("Running SecondIntegrationSpec")
    }
})

and the output is as expected:

Running UnitSpec
Starting fake embedded server...
Running FirstIntegrationSpec
Running SecondIntegrationSpec
Stopping fake embedded server...

[tibtof]

Small improvement to hide AfterProjectListener, that is not needed outside ProjectConfig:

object EmbeddedContainerExtension : BeforeSpecListener {
    val embeddedServer: FakeEmbeddedServer by lazy {
        FakeEmbeddedServer().apply { start() }
    }

    override suspend fun beforeSpec(spec: Spec) {
        embeddedServer
    }
}

object ProjectConfig : AbstractProjectConfig() {
    override fun extensions(): List<Extension> = listOf(object : AfterProjectListener {
        override suspend fun afterProject() {
            EmbeddedContainerExtension.embeddedServer.stop()
        }
    })
}

[tibtof]

And going back to the idea of providing this at tag level, maybe having tag level events could be an option:

fun interface BeforeTagListener : Extension {
    suspend fun beforeTag(tag: Tag)
}

object IT : Tag()

object EmbeddedContainerTagExtension : BeforeTagListener, AfterProjectListener {
    val embeddedServer: FakeEmbeddedServer by lazy {
        FakeEmbeddedServer().apply { start() }
    }

    override suspend fun beforeTag(tag: Tag) {
        if (tag == IT) {
            embeddedServer
        }
    }

    override suspend fun afterProject() {
        embeddedServer.stop()
    }

}

object ProjectTagConfig : AbstractProjectConfig() {
    override fun extensions(): List<Extension> = listOf(EmbeddedContainerTagExtension)
}

class AnUnitSpec : FreeSpec({
    "unit test" {
        println("Running UnitSpec")
    }
})

class FirstIntegrationSpec : FreeSpec({
    tags(IT)

    "using embedded server" {
        println("Running FirstIntegrationSpec")
    }

})

class SecondIntegrationSpec : FreeSpec({
    tags(IT)

    "using embedded server" {
        println("Running SecondIntegrationSpec")
    }
})

[tibtof]

Maybe something like:

val normalExtension = install(MyExtension(), shared = true)

In which case it shares with another spec and the shutdown logic is deferred to the end of the project.

A question would be if you did

val normalExtension = install(MyExtension(1), shared = true)

and

val normalExtension = install(MyExtension(2), shared = true)

Does it just ignore the 2nd one and return the first.

Would there be a problem if instead of having the shared flag we just reuse the same instance of the extension? e.g. val extension = install(MyExtension) where MyExtension is an object.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sksamuel]

There's SharedTestContainerExtension in the latest kotest test containers which will probably do what you want @tibtof

[tibtof]

@sksamuel thanks for the suggestion, SharedTestContainerExtension was exactly what I needed, but not enough for all cases. I also had some embedded servers that were not test containers. Thus I adapted SharedTestContainerExtension this way:

import io.kotest.core.extensions.MountableExtension
import io.kotest.core.listeners.AfterProjectListener
import io.kotest.core.listeners.AfterSpecListener
import io.kotest.core.listeners.AfterTestListener
import io.kotest.core.listeners.BeforeSpecListener
import io.kotest.core.listeners.BeforeTestListener
import io.kotest.core.spec.Spec
import io.kotest.core.test.TestCase
import io.kotest.core.test.TestResult

class SharedEmbeddedServerExtension<T, U>(
    private val embeddedServer: T,
    private val isRunning: T.() -> Boolean = { false },
    private val start: T.() -> Unit = {},
    private val stop: T.() -> Unit = {},
    private val beforeTest: suspend (T) -> Unit = {},
    private val afterTest: suspend (T) -> Unit = {},
    private val beforeSpec: suspend (T) -> Unit = {},
    private val afterSpec: suspend (T) -> Unit = {},
    private val configure: T.() -> Unit = {},
    private val mapper: T.() -> U,
) : MountableExtension<T, U>,
    AfterProjectListener,
    BeforeTestListener,
    BeforeSpecListener,
    AfterTestListener,
    AfterSpecListener {

    companion object {
        operator fun <T> invoke(
            embeddedServer: T,
            isRunning: T.() -> Boolean = { false },
            start: T.() -> Unit = {},
            stop: T.() -> Unit = {},
            beforeTest: (T) -> Unit = {},
            afterTest: (T) -> Unit = {},
            beforeSpec: (T) -> Unit = {},
            afterSpec: (T) -> Unit = {},
            configure: T.() -> Unit = {},
        ): SharedEmbeddedServerExtension<T, T> {
            return SharedEmbeddedServerExtension(
                embeddedServer,
                isRunning,
                start,
                stop,
                beforeTest,
                afterTest,
                beforeSpec,
                afterSpec,
                configure
            ) { this }
        }
    }

    override fun mount(configure: T.() -> Unit): U {
        if (!embeddedServer.isRunning()) {
            embeddedServer.start()
            configure(embeddedServer)
            [email protected](embeddedServer)
        }
        return [email protected](embeddedServer)
    }

    override suspend fun afterProject() {
        if (embeddedServer.isRunning()) embeddedServer.stop()
    }

    override suspend fun beforeTest(testCase: TestCase) {
        beforeTest(embeddedServer)
    }

    override suspend fun afterTest(testCase: TestCase, result: TestResult) {
        afterTest(embeddedServer)
    }

    override suspend fun beforeSpec(spec: Spec) {
        beforeSpec(embeddedServer)
    }

    override suspend fun afterSpec(spec: Spec) {
        afterSpec(embeddedServer)
    }
}

here's an example usage:

val wiremockExtension = SharedEmbeddedServerExtension(
    WireMockServer(
        WireMockConfiguration()
            .httpsPort(0)
    ),
    isRunning = WireMockServer::isRunning,
    start = WireMockServer::start,
    stop = WireMockServer::stop,
    afterTest = WireMockServer::resetAll
) {
    System.setProperty("mock.server.port", httpsPort().toString())
}

and in the test itself:

val wiremock = install(wiremockExtension)

wiremock.post {
    url equalTo "/my-endpoint"
} returns {
    //language=JSON
    body = """
        {
           "foo": "bar"
        }                               
    """.trimIndent()
}

I'm happy with this solution, now when I run just the unit tests the testcontainers and embedded servers are not started anymore, and the same instance of each server can be shared between multiple integration tests.