Create benchmarking infrastructure for JS (and Native?)

[armanbilge]

(This issue is not directly about Cats Effect, but highly relevant to it.)

The lack of benchmarks for JS came up again recently wrt https://github.com/typelevel/cats-effect/pull/2673. Daniel and I had a discussion about this on Discord and I wanted to write some notes/braindump.

Firstly, although there is scalajs-benchmark, it is really designed for making interactive browser benchmarks which is very cool, but introduces various complications and sub-optimalities for our use-case. I kept a fork going for a while in https://github.com/typelevel/cats-effect/pull/2229 but it was a hack-job.

So, my (crazy) proposal to benchmark JS is to shim JMH, similarly to how Scala.js itself shims JUnit.

1. Note that the JUnit shim had to add support for async tests, and we'd similarly have to make JMH support async benchmarks as well.
2. Since all the existing benchmarks in CE end in unsafeRunSync(), we can never achieve complete source-compatibility. It'll be annoying, but we can use a unsafeRunPlatform(...) method to share the benchmarks.
3. At first it seems like we need a mini-CE to implement this, but I actually don't think so. A benchmarking framework isn't some complicated concurrent program requiring fibers, cancellation etc. We can manage with MacrotaskExecutor and Future and if we can do without macrotask that would be even better because then we can use it to benchmark that too.
4. Although Scala.js doesn't directly support runtime annotations, it is possible to make them work. This is how JUnit.js works. IIUC it reads the annotations off of the corresponding .class files and does something with them.
5. Hotspot and V8 are very different things, so some concepts may not translate well. We'll have to think about these.

So the overall goals would be:

1. Source-compatibility with JMH, as much as possible. And similar output too for that matter.
2. Minimal dependency, pure Scala.js. No fiddling with JavaScript dependencies.
3. Ease-of-use directly from sbt. No fiddling with browser webpages.
4. Compatible with all the "real" JSEnvs: Node.js and browsers.
5. If it's not clear, this would be an independent project to CE of course :)

Why

• we can share the bulk of our existing benchmark suite with JVM/JS
• if it's easy enough to run, we'll do it more often, and learn how changes affect JS alongside JVM
• gives us the baseline to try experiments like a separate runloop for JS
• benchmark capabilities for JS can be interesting/helpful downstream in fs2, http4s, etc.
• oh, also maybe we'll make some meaningful optimizations :)

Why not

• major yak
• Performance can't be a first-class concern for both JVM and JS without splitting sources across platforms more frequently than we'd like. Since JVM rightfully takes priority, I'm not sure it's worth investing so much effort to create a benchmarking framework for JS.
• major yak

[armanbilge]

Btw, this effort can be re-used for Native (which also has shimmed JUnit like Scala.js did). Ideally this project could one day live under @portable-scala maybe.

[armanbilge]

Regretfully I've started shaving this yak. I am frustrated I cannot benchmark some of my recent Scala.js performance PRs. And I expect further frustration when we start to explore Native-specific optimizations.

https://github.com/armanbilge/scala-jmh

Let's see if it pans out. 😩