The benchmark penalizes languages that need to start a runtime environment (e.g. anything on the JVM or .Net)

[PEZ]

Hi,

Thanks for creating this project, @bddicken! 🙏

I have been running benchmarks in some different ways the last few days. I notice that a lot of the difference between the Java performance and C is about JVM start times. And for Clojure the penalty is much bigger, because then also Clojure needs to start before the benchmarks can start crunching. To “see” the start time I created a hello-world benchmark, where the programs only print “Hello World!”. Here's a run on my Mac M4:

Benchmarking C
Benchmark 1: ./c/code 40
  Time (mean ± σ):       2.2 ms ±   0.2 ms    [User: 0.7 ms, System: 1.2 ms]
  Range (min … max):     2.0 ms …   2.5 ms    10 runs
 

Benchmarking Java
Benchmark 1: java jvm.code 40
  Time (mean ± σ):      38.7 ms ±   0.9 ms    [User: 14.1 ms, System: 21.4 ms]
  Range (min … max):    37.3 ms …  40.2 ms    10 runs
 

Benchmarking Clojure
Benchmark 1: java -cp clojure/classes:src:/Users/pez/.m2/repository/org/clojure/clojure/1.12.0/clojure-1.12.0.jar:/Users/pez/.m2/repository/org/clojure/core.specs.alpha/0.4.74/core.specs.alpha-0.4.74.jar:/Users/pez/.m2/repository/org/clojure/spec.alpha/0.5.238/spec.alpha-0.5.238.jar code 40
  Time (mean ± σ):     275.0 ms ±   4.3 ms    [User: 410.4 ms, System: 43.4 ms]
  Range (min … max):   270.0 ms … 285.0 ms    10 runs
 

Benchmarking Babashka
Benchmark 1: bb -cp clojure -m code 40
  Time (mean ± σ):      11.9 ms ±   0.4 ms    [User: 3.3 ms, System: 6.1 ms]
  Range (min … max):    11.1 ms …  12.4 ms    10 runs

Basically the JVM takes 40ms to start, and then Clojure takes another 240ms. To be compared with the benchmark times for e.g. C, Java, and Clojure, which run at 250, 350, and 550 ms, respectively on the same machine.

I can submit the hello-world benchmark as a PR, if you're interested. Start times can be very important for e.g. shell scripting purposes and such. In the Clojure world we have Babashka which is interpreted Clojure. It starts super quickly, but of course can't compete with compiled languages in the other benchmarks.

Some questions:

• Is this something you have though any about?
• Would you be interested in me exploring ways to adapt the benchmark to give these VM-hosted languages a fairer run? Some ideas that so far has popped to my mind:
  • Include the hello-world benchmark and then use it to subtract the times there from the times in the other benchmarks.
  • Another is to add some way to use different benchmarking tools. So for the JVM, use some tool that runs the programs from inside the JVM.
  • Make the programs take a second argument specifying a number of runs. So the fibonacci 40 would be run x times, and with a significantly high x, the startup times would play much lesser role.
  • Extend the runs by running more loops and higher fibs.
  • Other than that I have no more ideas right now 😄 But am willing to start sourcing ideas and doing experiments and such.

WDYT?