Simplify creations of separate benchmark source sets in multiplatform projects

[fzhinkin]

trafficstars

Documentation describes a way to place benchmarks in separate source sets. It kinda works when you need a single source set associated with a particular target. However, creating a separate common benchmark source set where benchmarks could be executed for each and every target is quite difficult.

It would be nice to provide an extension function for Project, that'll do that itself, instead of delegating this tedious work to users.

[fzhinkin]

This is how "manual" configuration looks like now:

kotlin {
    sourceSets {
        val benchmarkMain by creating {
            dependencies {
                implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.13")
            }
        }
    }

    val bmm = sourceSets.getByName("benchmarkMain")

    targets.matching { it.name != "metadata" }.all {
        compilations.create("benchmark") {
            associateWith([email protected]("main"))
            defaultSourceSet {
                dependencies {
                    implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.13")
                }
                dependsOn(bmm)
            }
        }
    }

    targets.matching { it.name != "metadata" }.all {
        benchmark.targets.register("${name}Benchmark")
    }
}

Yes, it could be cleaned up and simplified. Yet, it's hard to come up with something like that unless you spend enough time writing and debugging build scripts.

[lppedd]

A good default for the plugin could be creating source sets based on active platforms. Example with js() and jvm():

commonMain
commonTest
commonBenchmark
jsMain
jsTest
jsBenchmark
jvmMain
jvmTest
jvmBenchmark

[cubuspl42]

I understand the idea of the word "benchmark" appearing in the second spot of this magical tuple. commonMain is the primary code, jvmMain is JVM-specific primary code (depends on commonMain), commonBenchmark is cross-platform benchmarking code, jvmBenchmark is JVM-specific benchmarking code (depends on commonBenchmark), etc...

But what's the role of benchmarkMain? Isn't commonBenchmark a good-enough greatest common ancestor of all benchmarking source sets?

[fzhinkin]

Yep, <srcset name>Benchmark seems to be a much better alternative to benchmarkMain.

[cubuspl42]

It's what you mentioned in the last comment. Do you know how could we "polyfill" exactly this setup?

[cubuspl42]

I found an official example for shared cross-platform benchmark code, but it's in commonMain, not including an optional sourceset separation for benchmarking code.

[fzhinkin]

Do you know how could we "polyfill" exactly this setup?

@cubuspl42, I guess something like https://github.com/Kotlin/kotlinx-benchmark/issues/291#issuecomment-2652276849 but with different source set names.