Is there an opportunity to run performance tests on suspend functions?

[JochenGuckSnk]

Hi Team,

i'm wondering if it is possible to do multi platform performance tests on suspended functions.

Example

    @Benchmark
    suspend fun test(): Int{
        return 1+1
    }

As a workaround i tried to use runTest from coroutines.

    val bh: Blackhole = createInCommon()

    @Benchmark
    fun test() = runTest{
        bh.consume(1+1)
    }

But it would need a common code Blackhole.

In addition i'm not sure if it will work on JS because runTest does not block.

Is there an alternative way of testing suspend code? Or are there plans to add the support to the plugin?

[qurbonzoda]

Hi @JochenGuckSnk, Could you please share what do you intend to benchmark or measure? And maybe you could also share how do you think suspend benchmark functions should work?

[whyoleg]

BTW, rsocket-kotlin benchmark suspend calls to test throughput of underlying implementation, and to compare it to java implementation in different scenarios. And there is the only way to test it, using suspend functions. https://github.com/rsocket/rsocket-kotlin/tree/master/benchmarks for reference For now it's only JVM, but I have a plans to introduce MPP benchmarks

[JochenGuckSnk]

Hi @qurbonzoda,

i'm working on a Kotlin multi platform middleware. Since there are software components which are running in a suspended context. The middleware needs to be suspended in large areas of its functionality. My goal is to benchmark the throughput of the middleware on JVM,JS and Native. It would be sufficient form my side if the underling measurements measure the blocking runtime. I just what an indicator of the load introduced by the middleware.

[qurbonzoda]

I am not sure how users would interpret:

@Benchmark
suspend fun f() {
    // call some suspend functions
}

IMHO explicit blocking with runBlocking looks concise enough:

@Benchmark
suspend fun f() = runBlocking {
    // call some suspend functions
}

Maybe you could provide examples where this approach is not that good?

[whyoleg]

My concerns:

1. runBlocking is not supported on JS :) So we need to create expect/actual for it similar that exists in kotlinx.coroutines
2. runBlocking still has some overhead, it would be cool, if it were single runBlocking per iteration(or configurable, per run?) instead of per benchmark function call

[whyoleg]

Regarding second point I've tested simple case of runBlocking overhead run benchmark with runBlocking per operation and run benchmark with runBlocking repeating 1000 operations inside it. so runBlocking { operation() } vs runBlocking repeat(1000) { operation() } } Here are results:

Benchmark                           (payloadParam)  (runnerParam)  (transportParam)   Mode  Cnt       Score       Error  Units
RequestResponseBenchmark.benchmark               0            B|_             local  thrpt    3  180362.456 ± 85738.425  ops/s
RequestResponseBenchmark.benchmark               0       R|_|1000             local  thrpt    3     211.505 ±    28.941  ops/s

So 180k vs 211k - 17% difference. The main problem is you need keep in mind, that result need to be multiplied by some CONSTANT, which differ from test to test, because some test can have 100000/s others 500/s and you can use 1000 or 10000 operations per benchmark in first case and 10/100 operation per benchmark for second case

Overall, benchmarks in my case used in 3 ways:

1. know overall max throughput of specific operation
2. compare different implementations (in my case rsocket-java vs rsocket-kotlin)
3. compare different versions of same implementation (f.e. rsocket-kotlin 0.15 vs 0.16)

only 1 case is affected by this runBlocking overhead, so I can leave with it :)

[JochenGuckSnk]

Thank you @whyoleg for the detailed analysis.

I just want to add that runBlocking for Js is a very long story distributed over many tickets: https://github.com/Kotlin/kotlinx.coroutines/issues/195

Since it seams to be hard to implement runBlocking for Js. Because of this runTest was implemented. But runTest has the shortcomings i mentioned. I checked the implementation shortly: https://github.com/Kotlin/kotlinx.coroutines/blob/3574c2feca23c3e8a1ad00b5bf92e2bf04d95060/kotlinx-coroutines-core/js/test/TestBase.kt For me it seams that the test is running in a promise but is not awaited. So the performance measurement will not measure the execution if the test code.