realm-kotlin

realm-kotlin copied to clipboard

Currently, the /benchmarks module has to run manually, but this should run as part of CI.

Requirements:

• [ ] Reproducible results with a predefined margin of error. This probably means we need to control the hardware. When moving to Github Actions, we can probably use Firebase testlab for Android performance tests, but it is unclear what to do about JVM tests.
• [ ] Be able to compare results against a baseline build and report the difference.
• [ ] Be able to collect and update baseline numbers in an easy and well-documented way.
• [ ] Be able to define what constitutes a failing performance test, e.g. how much did a certain metric change.

We probably want to be able to report and handle changes to min, max, average and certain percentiles (like 25, 50, 75, and 90).

Seems like Jetbrains created a Benchmark library that works on Kotlin Multiplatform and uses JMH under the hood: https://github.com/Kotlin/kotlinx-benchmark

It is probably worth experimenting with rather than using JMH directly on JVM as we do now.