Cobalt Replace interpreter with partial evaluation

Replace interpreter with partial evaluation

Open viluon opened this issue 4 years ago • 8 comments

Approach

This PR implements the first Futamura projection in Cobalt by replacing Lua bytecode with Java lambdas. Partial evaluation reduces branching on hot code paths and unlocks optimisation opportunities for both maintainers and JVM's JIT compiler.

Status

I've added changes and fixes to pass Cobalt's test suite. Now that it's all green I'll focus on optimising the hell out of this.

Performance comparison

This is the result of PerformanceBenchmark using the interpreter:

Benchmark                          Mode  Cnt  Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15  6.058 ± 0.578  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  9.478 ± 0.408  ops/s
PerformanceBenchmark.nbody        thrpt   15  1.083 ± 0.089  ops/s
PerformanceBenchmark.nsieve       thrpt   15  0.457 ± 0.028  ops/s

Here is the same benchmark executed with partial evaluation:

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.981 ± 0.191  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  11.092 ± 0.194  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.174 ± 0.024  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.524 ± 0.027  ops/s

The full output of both benchmark runs follows.

PerformanceBenchmark: interpretation

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=195708760018641 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42175:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=195708760018641 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42175:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 5.686 ops/s
# Warmup Iteration   2: 6.081 ops/s
# Warmup Iteration   3: 5.495 ops/s
# Warmup Iteration   4: 6.578 ops/s
# Warmup Iteration   5: 6.529 ops/s
Iteration   1: 6.353 ops/s
Iteration   2: 6.471 ops/s
Iteration   3: 5.531 ops/s
Iteration   4: 6.581 ops/s
Iteration   5: 4.880 ops/s

# Run progress: 8.33% complete, ETA 00:20:37
# Fork: 2 of 3
# Warmup Iteration   1: 6.355 ops/s
# Warmup Iteration   2: 6.194 ops/s
# Warmup Iteration   3: 6.309 ops/s
# Warmup Iteration   4: 6.632 ops/s
# Warmup Iteration   5: 6.553 ops/s
Iteration   1: 6.719 ops/s
Iteration   2: 6.810 ops/s
Iteration   3: 6.247 ops/s
Iteration   4: 5.877 ops/s
Iteration   5: 5.874 ops/s

# Run progress: 16.67% complete, ETA 00:18:39
# Fork: 3 of 3
# Warmup Iteration   1: 5.738 ops/s
# Warmup Iteration   2: 5.077 ops/s
# Warmup Iteration   3: 5.940 ops/s
# Warmup Iteration   4: 6.113 ops/s
# Warmup Iteration   5: 4.508 ops/s
Iteration   1: 5.187 ops/s
Iteration   2: 6.129 ops/s
Iteration   3: 5.947 ops/s
Iteration   4: 6.192 ops/s
Iteration   5: 6.075 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  6.058 ±(99.9%) 0.578 ops/s [Average]
  (min, avg, max) = (4.880, 6.058, 6.810), stdev = 0.541
  CI (99.9%): [5.480, 6.636] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=195708760018641 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42175:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:47
# Fork: 1 of 3
# Warmup Iteration   1: 9.089 ops/s
# Warmup Iteration   2: 9.484 ops/s
# Warmup Iteration   3: 8.556 ops/s
# Warmup Iteration   4: 9.179 ops/s
# Warmup Iteration   5: 9.730 ops/s
Iteration   1: 9.902 ops/s
Iteration   2: 9.892 ops/s
Iteration   3: 9.858 ops/s
Iteration   4: 9.742 ops/s
Iteration   5: 9.788 ops/s

# Run progress: 33.33% complete, ETA 00:14:54
# Fork: 2 of 3
# Warmup Iteration   1: 6.768 ops/s
# Warmup Iteration   2: 9.540 ops/s
# Warmup Iteration   3: 9.708 ops/s
# Warmup Iteration   4: 8.740 ops/s
# Warmup Iteration   5: 9.344 ops/s
Iteration   1: 9.388 ops/s
Iteration   2: 9.342 ops/s
Iteration   3: 8.928 ops/s
Iteration   4: 9.100 ops/s
Iteration   5: 8.803 ops/s

# Run progress: 41.67% complete, ETA 00:13:02
# Fork: 3 of 3
# Warmup Iteration   1: 8.039 ops/s
# Warmup Iteration   2: 9.082 ops/s
# Warmup Iteration   3: 9.519 ops/s
# Warmup Iteration   4: 9.196 ops/s
# Warmup Iteration   5: 9.472 ops/s
Iteration   1: 9.175 ops/s
Iteration   2: 9.858 ops/s
Iteration   3: 9.502 ops/s
Iteration   4: 9.125 ops/s
Iteration   5: 9.768 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  9.478 ±(99.9%) 0.408 ops/s [Average]
  (min, avg, max) = (8.803, 9.478, 9.902), stdev = 0.381
  CI (99.9%): [9.070, 9.886] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=195708760018641 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42175:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:10
# Fork: 1 of 3
# Warmup Iteration   1: 1.017 ops/s
# Warmup Iteration   2: 1.070 ops/s
# Warmup Iteration   3: 1.070 ops/s
# Warmup Iteration   4: 1.035 ops/s
# Warmup Iteration   5: 1.073 ops/s
Iteration   1: 0.971 ops/s
Iteration   2: 1.118 ops/s
Iteration   3: 1.072 ops/s
Iteration   4: 1.131 ops/s
Iteration   5: 1.127 ops/s

# Run progress: 58.33% complete, ETA 00:09:21
# Fork: 2 of 3
# Warmup Iteration   1: 1.060 ops/s
# Warmup Iteration   2: 1.096 ops/s
# Warmup Iteration   3: 1.107 ops/s
# Warmup Iteration   4: 1.028 ops/s
# Warmup Iteration   5: 1.032 ops/s
Iteration   1: 1.052 ops/s
Iteration   2: 1.049 ops/s
Iteration   3: 0.864 ops/s
Iteration   4: 1.068 ops/s
Iteration   5: 1.042 ops/s

# Run progress: 66.67% complete, ETA 00:07:30
# Fork: 3 of 3
# Warmup Iteration   1: 1.059 ops/s
# Warmup Iteration   2: 1.132 ops/s
# Warmup Iteration   3: 1.131 ops/s
# Warmup Iteration   4: 1.187 ops/s
# Warmup Iteration   5: 1.172 ops/s
Iteration   1: 1.136 ops/s
Iteration   2: 1.172 ops/s
Iteration   3: 1.137 ops/s
Iteration   4: 1.188 ops/s
Iteration   5: 1.117 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.083 ±(99.9%) 0.089 ops/s [Average]
  (min, avg, max) = (0.864, 1.083, 1.188), stdev = 0.083
  CI (99.9%): [0.994, 1.172] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=195708760018641 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42175:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:39
# Fork: 1 of 3
# Warmup Iteration   1: 0.407 ops/s
# Warmup Iteration   2: 0.405 ops/s
# Warmup Iteration   3: 0.433 ops/s
# Warmup Iteration   4: 0.410 ops/s
# Warmup Iteration   5: 0.466 ops/s
Iteration   1: 0.448 ops/s
Iteration   2: 0.406 ops/s
Iteration   3: 0.446 ops/s
Iteration   4: 0.435 ops/s
Iteration   5: 0.460 ops/s

# Run progress: 83.33% complete, ETA 00:03:48
# Fork: 2 of 3
# Warmup Iteration   1: 0.290 ops/s
# Warmup Iteration   2: 0.455 ops/s
# Warmup Iteration   3: 0.377 ops/s
# Warmup Iteration   4: 0.460 ops/s
# Warmup Iteration   5: 0.450 ops/s
Iteration   1: 0.485 ops/s
Iteration   2: 0.462 ops/s
Iteration   3: 0.459 ops/s
Iteration   4: 0.464 ops/s
Iteration   5: 0.404 ops/s

# Run progress: 91.67% complete, ETA 00:01:54
# Fork: 3 of 3
# Warmup Iteration   1: 0.380 ops/s
# Warmup Iteration   2: 0.452 ops/s
# Warmup Iteration   3: 0.428 ops/s
# Warmup Iteration   4: 0.426 ops/s
# Warmup Iteration   5: 0.480 ops/s
Iteration   1: 0.485 ops/s
Iteration   2: 0.488 ops/s
Iteration   3: 0.481 ops/s
Iteration   4: 0.481 ops/s
Iteration   5: 0.447 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.457 ±(99.9%) 0.028 ops/s [Average]
  (min, avg, max) = (0.404, 0.457, 0.488), stdev = 0.027
  CI (99.9%): [0.428, 0.485] (assumes normal distribution)


# Run complete. Total time: 00:23:00

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt  Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15  6.058 ± 0.578  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  9.478 ± 0.408  ops/s
PerformanceBenchmark.nbody        thrpt   15  1.083 ± 0.089  ops/s
PerformanceBenchmark.nsieve       thrpt   15  0.457 ± 0.028  ops/s

Process finished with exit code 0

PerformanceBenchmark: partial evaluation

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 4.120 ops/s
# Warmup Iteration   2: 6.120 ops/s
# Warmup Iteration   3: 7.043 ops/s
# Warmup Iteration   4: 6.247 ops/s
# Warmup Iteration   5: 6.892 ops/s
Iteration   1: 6.569 ops/s
Iteration   2: 6.965 ops/s
Iteration   3: 6.997 ops/s
Iteration   4: 6.916 ops/s
Iteration   5: 6.564 ops/s

# Run progress: 8.33% complete, ETA 00:20:43
# Fork: 2 of 3
# Warmup Iteration   1: 6.509 ops/s
# Warmup Iteration   2: 7.068 ops/s
# Warmup Iteration   3: 7.050 ops/s
# Warmup Iteration   4: 7.028 ops/s
# Warmup Iteration   5: 7.105 ops/s
Iteration   1: 6.992 ops/s
Iteration   2: 7.036 ops/s
Iteration   3: 7.132 ops/s
Iteration   4: 7.072 ops/s
Iteration   5: 7.133 ops/s

# Run progress: 16.67% complete, ETA 00:18:43
# Fork: 3 of 3
# Warmup Iteration   1: 6.499 ops/s
# Warmup Iteration   2: 7.004 ops/s
# Warmup Iteration   3: 6.988 ops/s
# Warmup Iteration   4: 7.031 ops/s
# Warmup Iteration   5: 6.995 ops/s
Iteration   1: 7.038 ops/s
Iteration   2: 7.081 ops/s
Iteration   3: 7.039 ops/s
Iteration   4: 7.083 ops/s
Iteration   5: 7.100 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  6.981 ±(99.9%) 0.191 ops/s [Average]
  (min, avg, max) = (6.564, 6.981, 7.133), stdev = 0.179
  CI (99.9%): [6.790, 7.172] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:48
# Fork: 1 of 3
# Warmup Iteration   1: 10.260 ops/s
# Warmup Iteration   2: 11.155 ops/s
# Warmup Iteration   3: 11.222 ops/s
# Warmup Iteration   4: 10.839 ops/s
# Warmup Iteration   5: 11.186 ops/s
Iteration   1: 11.332 ops/s
Iteration   2: 10.889 ops/s
Iteration   3: 10.825 ops/s
Iteration   4: 11.210 ops/s
Iteration   5: 11.364 ops/s

# Run progress: 33.33% complete, ETA 00:14:55
# Fork: 2 of 3
# Warmup Iteration   1: 10.037 ops/s
# Warmup Iteration   2: 10.824 ops/s
# Warmup Iteration   3: 11.103 ops/s
# Warmup Iteration   4: 11.123 ops/s
# Warmup Iteration   5: 11.050 ops/s
Iteration   1: 11.140 ops/s
Iteration   2: 10.830 ops/s
Iteration   3: 11.122 ops/s
Iteration   4: 11.089 ops/s
Iteration   5: 11.217 ops/s

# Run progress: 41.67% complete, ETA 00:13:02
# Fork: 3 of 3
# Warmup Iteration   1: 10.092 ops/s
# Warmup Iteration   2: 11.080 ops/s
# Warmup Iteration   3: 10.915 ops/s
# Warmup Iteration   4: 11.093 ops/s
# Warmup Iteration   5: 11.146 ops/s
Iteration   1: 11.149 ops/s
Iteration   2: 11.176 ops/s
Iteration   3: 10.900 ops/s
Iteration   4: 10.884 ops/s
Iteration   5: 11.248 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  11.092 ±(99.9%) 0.194 ops/s [Average]
  (min, avg, max) = (10.825, 11.092, 11.364), stdev = 0.181
  CI (99.9%): [10.898, 11.286] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:10
# Fork: 1 of 3
# Warmup Iteration   1: 1.125 ops/s
# Warmup Iteration   2: 1.180 ops/s
# Warmup Iteration   3: 1.182 ops/s
# Warmup Iteration   4: 1.164 ops/s
# Warmup Iteration   5: 1.168 ops/s
Iteration   1: 1.178 ops/s
Iteration   2: 1.167 ops/s
Iteration   3: 1.192 ops/s
Iteration   4: 1.173 ops/s
Iteration   5: 1.182 ops/s

# Run progress: 58.33% complete, ETA 00:09:21
# Fork: 2 of 3
# Warmup Iteration   1: 1.128 ops/s
# Warmup Iteration   2: 1.123 ops/s
# Warmup Iteration   3: 1.141 ops/s
# Warmup Iteration   4: 1.152 ops/s
# Warmup Iteration   5: 1.092 ops/s
Iteration   1: 1.153 ops/s
Iteration   2: 1.146 ops/s
Iteration   3: 1.156 ops/s
Iteration   4: 1.152 ops/s
Iteration   5: 1.136 ops/s

# Run progress: 66.67% complete, ETA 00:07:30
# Fork: 3 of 3
# Warmup Iteration   1: 1.138 ops/s
# Warmup Iteration   2: 1.182 ops/s
# Warmup Iteration   3: 1.196 ops/s
# Warmup Iteration   4: 1.199 ops/s
# Warmup Iteration   5: 1.192 ops/s
Iteration   1: 1.191 ops/s
Iteration   2: 1.198 ops/s
Iteration   3: 1.186 ops/s
Iteration   4: 1.214 ops/s
Iteration   5: 1.187 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.174 ±(99.9%) 0.024 ops/s [Average]
  (min, avg, max) = (1.136, 1.174, 1.214), stdev = 0.022
  CI (99.9%): [1.151, 1.198] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:38
# Fork: 1 of 3
# Warmup Iteration   1: 0.484 ops/s
# Warmup Iteration   2: 0.509 ops/s
# Warmup Iteration   3: 0.509 ops/s
# Warmup Iteration   4: 0.519 ops/s
# Warmup Iteration   5: 0.493 ops/s
Iteration   1: 0.455 ops/s
Iteration   2: 0.528 ops/s
Iteration   3: 0.530 ops/s
Iteration   4: 0.517 ops/s
Iteration   5: 0.480 ops/s

# Run progress: 83.33% complete, ETA 00:03:47
# Fork: 2 of 3
# Warmup Iteration   1: 0.482 ops/s
# Warmup Iteration   2: 0.495 ops/s
# Warmup Iteration   3: 0.547 ops/s
# Warmup Iteration   4: 0.546 ops/s
# Warmup Iteration   5: 0.514 ops/s
Iteration   1: 0.527 ops/s
Iteration   2: 0.549 ops/s
Iteration   3: 0.548 ops/s
Iteration   4: 0.553 ops/s
Iteration   5: 0.523 ops/s

# Run progress: 91.67% complete, ETA 00:01:54
# Fork: 3 of 3
# Warmup Iteration   1: 0.426 ops/s
# Warmup Iteration   2: 0.517 ops/s
# Warmup Iteration   3: 0.527 ops/s
# Warmup Iteration   4: 0.511 ops/s
# Warmup Iteration   5: 0.533 ops/s
Iteration   1: 0.527 ops/s
Iteration   2: 0.531 ops/s
Iteration   3: 0.532 ops/s
Iteration   4: 0.535 ops/s
Iteration   5: 0.529 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.524 ±(99.9%) 0.027 ops/s [Average]
  (min, avg, max) = (0.455, 0.524, 0.553), stdev = 0.025
  CI (99.9%): [0.497, 0.551] (assumes normal distribution)


# Run complete. Total time: 00:23:01

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.981 ± 0.191  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  11.092 ± 0.194  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.174 ± 0.024  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.524 ± 0.027  ops/s

Process finished with exit code 0

May 07 '20 20:05 viluon

Gotta love how broken GH actions is at times.

May 09 '20 11:05 SquidDev

As of commit https://github.com/SquidDev/Cobalt/pull/38/commits/5fd9b71ca1af2713735a54c150fdd3a61d4c5130, the performance is as follows:

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.981 ± 0.191  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  11.092 ± 0.194  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.174 ± 0.024  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.524 ± 0.027  ops/s

Still faster than master, but not by nearly as much as before. More optimisations are on the way. For completeness' sake, I upgraded the kernel since opening this PR.

I'm marking this PR as "ready for review" -- more work has yet to be done, but now is the right time to dive into the code and point out potential problems which the test suite failed to catch.

As usual, here's the full benchmark log.

PerformanceBenchmark

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 4.120 ops/s
# Warmup Iteration   2: 6.120 ops/s
# Warmup Iteration   3: 7.043 ops/s
# Warmup Iteration   4: 6.247 ops/s
# Warmup Iteration   5: 6.892 ops/s
Iteration   1: 6.569 ops/s
Iteration   2: 6.965 ops/s
Iteration   3: 6.997 ops/s
Iteration   4: 6.916 ops/s
Iteration   5: 6.564 ops/s

# Run progress: 8.33% complete, ETA 00:20:43
# Fork: 2 of 3
# Warmup Iteration   1: 6.509 ops/s
# Warmup Iteration   2: 7.068 ops/s
# Warmup Iteration   3: 7.050 ops/s
# Warmup Iteration   4: 7.028 ops/s
# Warmup Iteration   5: 7.105 ops/s
Iteration   1: 6.992 ops/s
Iteration   2: 7.036 ops/s
Iteration   3: 7.132 ops/s
Iteration   4: 7.072 ops/s
Iteration   5: 7.133 ops/s

# Run progress: 16.67% complete, ETA 00:18:43
# Fork: 3 of 3
# Warmup Iteration   1: 6.499 ops/s
# Warmup Iteration   2: 7.004 ops/s
# Warmup Iteration   3: 6.988 ops/s
# Warmup Iteration   4: 7.031 ops/s
# Warmup Iteration   5: 6.995 ops/s
Iteration   1: 7.038 ops/s
Iteration   2: 7.081 ops/s
Iteration   3: 7.039 ops/s
Iteration   4: 7.083 ops/s
Iteration   5: 7.100 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  6.981 ±(99.9%) 0.191 ops/s [Average]
  (min, avg, max) = (6.564, 6.981, 7.133), stdev = 0.179
  CI (99.9%): [6.790, 7.172] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:48
# Fork: 1 of 3
# Warmup Iteration   1: 10.260 ops/s
# Warmup Iteration   2: 11.155 ops/s
# Warmup Iteration   3: 11.222 ops/s
# Warmup Iteration   4: 10.839 ops/s
# Warmup Iteration   5: 11.186 ops/s
Iteration   1: 11.332 ops/s
Iteration   2: 10.889 ops/s
Iteration   3: 10.825 ops/s
Iteration   4: 11.210 ops/s
Iteration   5: 11.364 ops/s

# Run progress: 33.33% complete, ETA 00:14:55
# Fork: 2 of 3
# Warmup Iteration   1: 10.037 ops/s
# Warmup Iteration   2: 10.824 ops/s
# Warmup Iteration   3: 11.103 ops/s
# Warmup Iteration   4: 11.123 ops/s
# Warmup Iteration   5: 11.050 ops/s
Iteration   1: 11.140 ops/s
Iteration   2: 10.830 ops/s
Iteration   3: 11.122 ops/s
Iteration   4: 11.089 ops/s
Iteration   5: 11.217 ops/s

# Run progress: 41.67% complete, ETA 00:13:02
# Fork: 3 of 3
# Warmup Iteration   1: 10.092 ops/s
# Warmup Iteration   2: 11.080 ops/s
# Warmup Iteration   3: 10.915 ops/s
# Warmup Iteration   4: 11.093 ops/s
# Warmup Iteration   5: 11.146 ops/s
Iteration   1: 11.149 ops/s
Iteration   2: 11.176 ops/s
Iteration   3: 10.900 ops/s
Iteration   4: 10.884 ops/s
Iteration   5: 11.248 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  11.092 ±(99.9%) 0.194 ops/s [Average]
  (min, avg, max) = (10.825, 11.092, 11.364), stdev = 0.181
  CI (99.9%): [10.898, 11.286] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:10
# Fork: 1 of 3
# Warmup Iteration   1: 1.125 ops/s
# Warmup Iteration   2: 1.180 ops/s
# Warmup Iteration   3: 1.182 ops/s
# Warmup Iteration   4: 1.164 ops/s
# Warmup Iteration   5: 1.168 ops/s
Iteration   1: 1.178 ops/s
Iteration   2: 1.167 ops/s
Iteration   3: 1.192 ops/s
Iteration   4: 1.173 ops/s
Iteration   5: 1.182 ops/s

# Run progress: 58.33% complete, ETA 00:09:21
# Fork: 2 of 3
# Warmup Iteration   1: 1.128 ops/s
# Warmup Iteration   2: 1.123 ops/s
# Warmup Iteration   3: 1.141 ops/s
# Warmup Iteration   4: 1.152 ops/s
# Warmup Iteration   5: 1.092 ops/s
Iteration   1: 1.153 ops/s
Iteration   2: 1.146 ops/s
Iteration   3: 1.156 ops/s
Iteration   4: 1.152 ops/s
Iteration   5: 1.136 ops/s

# Run progress: 66.67% complete, ETA 00:07:30
# Fork: 3 of 3
# Warmup Iteration   1: 1.138 ops/s
# Warmup Iteration   2: 1.182 ops/s
# Warmup Iteration   3: 1.196 ops/s
# Warmup Iteration   4: 1.199 ops/s
# Warmup Iteration   5: 1.192 ops/s
Iteration   1: 1.191 ops/s
Iteration   2: 1.198 ops/s
Iteration   3: 1.186 ops/s
Iteration   4: 1.214 ops/s
Iteration   5: 1.187 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.174 ±(99.9%) 0.024 ops/s [Average]
  (min, avg, max) = (1.136, 1.174, 1.214), stdev = 0.022
  CI (99.9%): [1.151, 1.198] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=4806246229622 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=44757:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:38
# Fork: 1 of 3
# Warmup Iteration   1: 0.484 ops/s
# Warmup Iteration   2: 0.509 ops/s
# Warmup Iteration   3: 0.509 ops/s
# Warmup Iteration   4: 0.519 ops/s
# Warmup Iteration   5: 0.493 ops/s
Iteration   1: 0.455 ops/s
Iteration   2: 0.528 ops/s
Iteration   3: 0.530 ops/s
Iteration   4: 0.517 ops/s
Iteration   5: 0.480 ops/s

# Run progress: 83.33% complete, ETA 00:03:47
# Fork: 2 of 3
# Warmup Iteration   1: 0.482 ops/s
# Warmup Iteration   2: 0.495 ops/s
# Warmup Iteration   3: 0.547 ops/s
# Warmup Iteration   4: 0.546 ops/s
# Warmup Iteration   5: 0.514 ops/s
Iteration   1: 0.527 ops/s
Iteration   2: 0.549 ops/s
Iteration   3: 0.548 ops/s
Iteration   4: 0.553 ops/s
Iteration   5: 0.523 ops/s

# Run progress: 91.67% complete, ETA 00:01:54
# Fork: 3 of 3
# Warmup Iteration   1: 0.426 ops/s
# Warmup Iteration   2: 0.517 ops/s
# Warmup Iteration   3: 0.527 ops/s
# Warmup Iteration   4: 0.511 ops/s
# Warmup Iteration   5: 0.533 ops/s
Iteration   1: 0.527 ops/s
Iteration   2: 0.531 ops/s
Iteration   3: 0.532 ops/s
Iteration   4: 0.535 ops/s
Iteration   5: 0.529 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.524 ±(99.9%) 0.027 ops/s [Average]
  (min, avg, max) = (0.455, 0.524, 0.553), stdev = 0.025
  CI (99.9%): [0.497, 0.551] (assumes normal distribution)


# Run complete. Total time: 00:23:01

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.981 ± 0.191  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  11.092 ± 0.194  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.174 ± 0.024  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.524 ± 0.027  ops/s

Process finished with exit code 0

May 09 '20 11:05 viluon

This is looking really nice, thank you. I'll try to have a fiddle later today, maybe load the logs within JITWatch and see if there's anything obvious which is problematic.

Just left a couple of comments and questions about some of the changes. I might cherry-pick out the Print.java changes on to master now, just to make this changeset cleaner.

May 09 '20 12:05 SquidDev

The advertised refactor of EvalCont is complete, here are the results:

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.993 ± 0.152  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  11.938 ± 0.084  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.153 ± 0.010  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.569 ± 0.013  ops/s

This is slightly faster than the previous benchmark in all cases except for nbody, where the new result lies within the error bar of the previous one.

PerformanceBenchmark output

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=16456715958750 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=45239:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=16456715958750 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=45239:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 6.365 ops/s
# Warmup Iteration   2: 6.776 ops/s
# Warmup Iteration   3: 6.932 ops/s
# Warmup Iteration   4: 6.913 ops/s
# Warmup Iteration   5: 6.949 ops/s
Iteration   1: 6.959 ops/s
Iteration   2: 6.948 ops/s
Iteration   3: 6.967 ops/s
Iteration   4: 7.036 ops/s
Iteration   5: 6.936 ops/s

# Run progress: 8.33% complete, ETA 00:20:31
# Fork: 2 of 3
# Warmup Iteration   1: 6.739 ops/s
# Warmup Iteration   2: 7.013 ops/s
# Warmup Iteration   3: 6.872 ops/s
# Warmup Iteration   4: 6.998 ops/s
# Warmup Iteration   5: 6.972 ops/s
Iteration   1: 7.013 ops/s
Iteration   2: 7.064 ops/s
Iteration   3: 7.046 ops/s
Iteration   4: 6.947 ops/s
Iteration   5: 6.531 ops/s

# Run progress: 16.67% complete, ETA 00:18:37
# Fork: 3 of 3
# Warmup Iteration   1: 6.759 ops/s
# Warmup Iteration   2: 7.089 ops/s
# Warmup Iteration   3: 6.978 ops/s
# Warmup Iteration   4: 6.987 ops/s
# Warmup Iteration   5: 7.086 ops/s
Iteration   1: 7.065 ops/s
Iteration   2: 7.072 ops/s
Iteration   3: 7.106 ops/s
Iteration   4: 7.107 ops/s
Iteration   5: 7.100 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  6.993 ±(99.9%) 0.152 ops/s [Average]
  (min, avg, max) = (6.531, 6.993, 7.107), stdev = 0.142
  CI (99.9%): [6.842, 7.145] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=16456715958750 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=45239:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:44
# Fork: 1 of 3
# Warmup Iteration   1: 11.026 ops/s
# Warmup Iteration   2: 11.840 ops/s
# Warmup Iteration   3: 11.819 ops/s
# Warmup Iteration   4: 11.757 ops/s
# Warmup Iteration   5: 12.019 ops/s
Iteration   1: 11.879 ops/s
Iteration   2: 11.957 ops/s
Iteration   3: 11.954 ops/s
Iteration   4: 11.931 ops/s
Iteration   5: 11.995 ops/s

# Run progress: 33.33% complete, ETA 00:14:52
# Fork: 2 of 3
# Warmup Iteration   1: 11.172 ops/s
# Warmup Iteration   2: 11.127 ops/s
# Warmup Iteration   3: 11.124 ops/s
# Warmup Iteration   4: 11.844 ops/s
# Warmup Iteration   5: 11.889 ops/s
Iteration   1: 12.023 ops/s
Iteration   2: 11.821 ops/s
Iteration   3: 12.025 ops/s
Iteration   4: 11.995 ops/s
Iteration   5: 11.745 ops/s

# Run progress: 41.67% complete, ETA 00:13:00
# Fork: 3 of 3
# Warmup Iteration   1: 10.987 ops/s
# Warmup Iteration   2: 11.790 ops/s
# Warmup Iteration   3: 11.861 ops/s
# Warmup Iteration   4: 11.699 ops/s
# Warmup Iteration   5: 11.811 ops/s
Iteration   1: 12.022 ops/s
Iteration   2: 11.973 ops/s
Iteration   3: 11.909 ops/s
Iteration   4: 11.899 ops/s
Iteration   5: 11.949 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  11.938 ±(99.9%) 0.084 ops/s [Average]
  (min, avg, max) = (11.745, 11.938, 12.025), stdev = 0.079
  CI (99.9%): [11.854, 12.023] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=16456715958750 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=45239:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:08
# Fork: 1 of 3
# Warmup Iteration   1: 1.113 ops/s
# Warmup Iteration   2: 1.132 ops/s
# Warmup Iteration   3: 1.152 ops/s
# Warmup Iteration   4: 1.151 ops/s
# Warmup Iteration   5: 1.153 ops/s
Iteration   1: 1.142 ops/s
Iteration   2: 1.145 ops/s
Iteration   3: 1.143 ops/s
Iteration   4: 1.144 ops/s
Iteration   5: 1.151 ops/s

# Run progress: 58.33% complete, ETA 00:09:19
# Fork: 2 of 3
# Warmup Iteration   1: 1.101 ops/s
# Warmup Iteration   2: 1.140 ops/s
# Warmup Iteration   3: 1.152 ops/s
# Warmup Iteration   4: 1.142 ops/s
# Warmup Iteration   5: 1.161 ops/s
Iteration   1: 1.158 ops/s
Iteration   2: 1.141 ops/s
Iteration   3: 1.166 ops/s
Iteration   4: 1.153 ops/s
Iteration   5: 1.155 ops/s

# Run progress: 66.67% complete, ETA 00:07:28
# Fork: 3 of 3
# Warmup Iteration   1: 1.133 ops/s
# Warmup Iteration   2: 1.129 ops/s
# Warmup Iteration   3: 1.156 ops/s
# Warmup Iteration   4: 1.164 ops/s
# Warmup Iteration   5: 1.157 ops/s
Iteration   1: 1.168 ops/s
Iteration   2: 1.163 ops/s
Iteration   3: 1.149 ops/s
Iteration   4: 1.162 ops/s
Iteration   5: 1.162 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.153 ±(99.9%) 0.010 ops/s [Average]
  (min, avg, max) = (1.141, 1.153, 1.168), stdev = 0.009
  CI (99.9%): [1.144, 1.163] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=16456715958750 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=45239:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:37
# Fork: 1 of 3
# Warmup Iteration   1: 0.547 ops/s
# Warmup Iteration   2: 0.581 ops/s
# Warmup Iteration   3: 0.573 ops/s
# Warmup Iteration   4: 0.586 ops/s
# Warmup Iteration   5: 0.577 ops/s
Iteration   1: 0.574 ops/s
Iteration   2: 0.591 ops/s
Iteration   3: 0.556 ops/s
Iteration   4: 0.585 ops/s
Iteration   5: 0.584 ops/s

# Run progress: 83.33% complete, ETA 00:03:46
# Fork: 2 of 3
# Warmup Iteration   1: 0.530 ops/s
# Warmup Iteration   2: 0.560 ops/s
# Warmup Iteration   3: 0.556 ops/s
# Warmup Iteration   4: 0.564 ops/s
# Warmup Iteration   5: 0.567 ops/s
Iteration   1: 0.561 ops/s
Iteration   2: 0.575 ops/s
Iteration   3: 0.568 ops/s
Iteration   4: 0.550 ops/s
Iteration   5: 0.573 ops/s

# Run progress: 91.67% complete, ETA 00:01:53
# Fork: 3 of 3
# Warmup Iteration   1: 0.532 ops/s
# Warmup Iteration   2: 0.559 ops/s
# Warmup Iteration   3: 0.551 ops/s
# Warmup Iteration   4: 0.557 ops/s
# Warmup Iteration   5: 0.567 ops/s
Iteration   1: 0.567 ops/s
Iteration   2: 0.568 ops/s
Iteration   3: 0.569 ops/s
Iteration   4: 0.546 ops/s
Iteration   5: 0.566 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.569 ±(99.9%) 0.013 ops/s [Average]
  (min, avg, max) = (0.546, 0.569, 0.591), stdev = 0.013
  CI (99.9%): [0.556, 0.582] (assumes normal distribution)


# Run complete. Total time: 00:22:44

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.993 ± 0.152  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  11.938 ± 0.084  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.153 ± 0.010  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.569 ± 0.013  ops/s

Process finished with exit code 0

May 09 '20 15:05 viluon

So I ran a the benchmarks over all commits up until master. I love that even commits which shouldn't have changed anything still show up on the benchmarks, which suggests this really isn't as reliable as I'd like. There's still a definite improvement over master though

May 10 '20 08:05 SquidDev

https://github.com/SquidDev/Cobalt/pull/38/commits/ac6810d2125b32c10fcc726f25966d4598e989fc is once again a controversial change. It seems to mildly improve performance of nbody and nsieve but performs worse with binarytrees and significantly worse with fannkuch. This is quite puzzling, as -- to the best of my knowledge -- the commit doesn't introduce any changes which would noticeably impact performance, at least not in a negative way.

I had chromium, thunderbird, and telegram open during this benchmark run, but I wasn't using the computer, so the CPU was pretty idle.

Benchmark                          Mode  Cnt  Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15  6.710 ± 0.147  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  9.968 ± 0.378  ops/s
PerformanceBenchmark.nbody        thrpt   15  1.213 ± 0.015  ops/s
PerformanceBenchmark.nsieve       thrpt   15  0.571 ± 0.008  ops/s

Nevertheless, I ran the benchmark again, this time with no other resource-intensive programs in the background.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.697 ± 0.089  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  10.201 ± 0.072  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.229 ± 0.019  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.592 ± 0.010  ops/s

PerformanceBenchmark: regular scenario

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=5131604619298 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=36101:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=5131604619298 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=36101:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 5.986 ops/s
# Warmup Iteration   2: 6.642 ops/s
# Warmup Iteration   3: 6.580 ops/s
# Warmup Iteration   4: 6.684 ops/s
# Warmup Iteration   5: 6.682 ops/s
Iteration   1: 6.735 ops/s
Iteration   2: 6.740 ops/s
Iteration   3: 6.740 ops/s
Iteration   4: 6.732 ops/s
Iteration   5: 6.691 ops/s

# Run progress: 8.33% complete, ETA 00:20:28
# Fork: 2 of 3
# Warmup Iteration   1: 6.373 ops/s
# Warmup Iteration   2: 6.465 ops/s
# Warmup Iteration   3: 6.530 ops/s
# Warmup Iteration   4: 6.556 ops/s
# Warmup Iteration   5: 6.492 ops/s
Iteration   1: 6.522 ops/s
Iteration   2: 6.547 ops/s
Iteration   3: 6.537 ops/s
Iteration   4: 6.550 ops/s
Iteration   5: 6.560 ops/s

# Run progress: 16.67% complete, ETA 00:18:36
# Fork: 3 of 3
# Warmup Iteration   1: 6.509 ops/s
# Warmup Iteration   2: 6.847 ops/s
# Warmup Iteration   3: 6.773 ops/s
# Warmup Iteration   4: 6.881 ops/s
# Warmup Iteration   5: 6.913 ops/s
Iteration   1: 6.878 ops/s
Iteration   2: 6.898 ops/s
Iteration   3: 6.846 ops/s
Iteration   4: 6.891 ops/s
Iteration   5: 6.788 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  6.710 ±(99.9%) 0.147 ops/s [Average]
  (min, avg, max) = (6.522, 6.710, 6.898), stdev = 0.137
  CI (99.9%): [6.563, 6.857] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=5131604619298 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=36101:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:44
# Fork: 1 of 3
# Warmup Iteration   1: 9.252 ops/s
# Warmup Iteration   2: 9.949 ops/s
# Warmup Iteration   3: 9.971 ops/s
# Warmup Iteration   4: 9.896 ops/s
# Warmup Iteration   5: 10.064 ops/s
Iteration   1: 10.009 ops/s
Iteration   2: 10.082 ops/s
Iteration   3: 10.109 ops/s
Iteration   4: 10.051 ops/s
Iteration   5: 10.010 ops/s

# Run progress: 33.33% complete, ETA 00:14:52
# Fork: 2 of 3
# Warmup Iteration   1: 9.446 ops/s
# Warmup Iteration   2: 10.088 ops/s
# Warmup Iteration   3: 10.044 ops/s
# Warmup Iteration   4: 10.013 ops/s
# Warmup Iteration   5: 10.125 ops/s
Iteration   1: 9.648 ops/s
Iteration   2: 8.766 ops/s
Iteration   3: 10.136 ops/s
Iteration   4: 10.044 ops/s
Iteration   5: 10.067 ops/s

# Run progress: 41.67% complete, ETA 00:13:00
# Fork: 3 of 3
# Warmup Iteration   1: 9.337 ops/s
# Warmup Iteration   2: 9.904 ops/s
# Warmup Iteration   3: 10.131 ops/s
# Warmup Iteration   4: 10.095 ops/s
# Warmup Iteration   5: 10.024 ops/s
Iteration   1: 10.172 ops/s
Iteration   2: 10.131 ops/s
Iteration   3: 10.073 ops/s
Iteration   4: 10.091 ops/s
Iteration   5: 10.128 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  9.968 ±(99.9%) 0.378 ops/s [Average]
  (min, avg, max) = (8.766, 9.968, 10.172), stdev = 0.354
  CI (99.9%): [9.589, 10.346] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=5131604619298 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=36101:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:08
# Fork: 1 of 3
# Warmup Iteration   1: 1.181 ops/s
# Warmup Iteration   2: 1.183 ops/s
# Warmup Iteration   3: 1.209 ops/s
# Warmup Iteration   4: 1.214 ops/s
# Warmup Iteration   5: 1.205 ops/s
Iteration   1: 1.196 ops/s
Iteration   2: 1.222 ops/s
Iteration   3: 1.205 ops/s
Iteration   4: 1.204 ops/s
Iteration   5: 1.208 ops/s

# Run progress: 58.33% complete, ETA 00:09:20
# Fork: 2 of 3
# Warmup Iteration   1: 1.160 ops/s
# Warmup Iteration   2: 1.183 ops/s
# Warmup Iteration   3: 1.195 ops/s
# Warmup Iteration   4: 1.203 ops/s
# Warmup Iteration   5: 1.193 ops/s
Iteration   1: 1.203 ops/s
Iteration   2: 1.191 ops/s
Iteration   3: 1.209 ops/s
Iteration   4: 1.200 ops/s
Iteration   5: 1.210 ops/s

# Run progress: 66.67% complete, ETA 00:07:29
# Fork: 3 of 3
# Warmup Iteration   1: 1.184 ops/s
# Warmup Iteration   2: 1.212 ops/s
# Warmup Iteration   3: 1.229 ops/s
# Warmup Iteration   4: 1.237 ops/s
# Warmup Iteration   5: 1.241 ops/s
Iteration   1: 1.224 ops/s
Iteration   2: 1.233 ops/s
Iteration   3: 1.232 ops/s
Iteration   4: 1.233 ops/s
Iteration   5: 1.225 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.213 ±(99.9%) 0.015 ops/s [Average]
  (min, avg, max) = (1.191, 1.213, 1.233), stdev = 0.014
  CI (99.9%): [1.198, 1.228] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=5131604619298 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=36101:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:37
# Fork: 1 of 3
# Warmup Iteration   1: 0.519 ops/s
# Warmup Iteration   2: 0.553 ops/s
# Warmup Iteration   3: 0.577 ops/s
# Warmup Iteration   4: 0.548 ops/s
# Warmup Iteration   5: 0.561 ops/s
Iteration   1: 0.574 ops/s
Iteration   2: 0.568 ops/s
Iteration   3: 0.575 ops/s
Iteration   4: 0.561 ops/s
Iteration   5: 0.562 ops/s

# Run progress: 83.33% complete, ETA 00:03:45
# Fork: 2 of 3
# Warmup Iteration   1: 0.530 ops/s
# Warmup Iteration   2: 0.564 ops/s
# Warmup Iteration   3: 0.559 ops/s
# Warmup Iteration   4: 0.564 ops/s
# Warmup Iteration   5: 0.569 ops/s
Iteration   1: 0.568 ops/s
Iteration   2: 0.568 ops/s
Iteration   3: 0.561 ops/s
Iteration   4: 0.572 ops/s
Iteration   5: 0.567 ops/s

# Run progress: 91.67% complete, ETA 00:01:53
# Fork: 3 of 3
# Warmup Iteration   1: 0.526 ops/s
# Warmup Iteration   2: 0.575 ops/s
# Warmup Iteration   3: 0.587 ops/s
# Warmup Iteration   4: 0.554 ops/s
# Warmup Iteration   5: 0.587 ops/s
Iteration   1: 0.582 ops/s
Iteration   2: 0.581 ops/s
Iteration   3: 0.580 ops/s
Iteration   4: 0.581 ops/s
Iteration   5: 0.571 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.571 ±(99.9%) 0.008 ops/s [Average]
  (min, avg, max) = (0.561, 0.571, 0.582), stdev = 0.007
  CI (99.9%): [0.564, 0.579] (assumes normal distribution)


# Run complete. Total time: 00:22:42

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt  Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15  6.710 ± 0.147  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  9.968 ± 0.378  ops/s
PerformanceBenchmark.nbody        thrpt   15  1.213 ± 0.015  ops/s
PerformanceBenchmark.nsieve       thrpt   15  0.571 ± 0.008  ops/s

Process finished with exit code 0

PerformanceBenchmark: few other programs running

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=7277273239412 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42307:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=7277273239412 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42307:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 6.335 ops/s
# Warmup Iteration   2: 6.672 ops/s
# Warmup Iteration   3: 6.725 ops/s
# Warmup Iteration   4: 6.743 ops/s
# Warmup Iteration   5: 6.734 ops/s
Iteration   1: 6.744 ops/s
Iteration   2: 6.720 ops/s
Iteration   3: 6.784 ops/s
Iteration   4: 6.803 ops/s
Iteration   5: 6.727 ops/s

# Run progress: 8.33% complete, ETA 00:20:30
# Fork: 2 of 3
# Warmup Iteration   1: 6.456 ops/s
# Warmup Iteration   2: 6.497 ops/s
# Warmup Iteration   3: 6.527 ops/s
# Warmup Iteration   4: 6.549 ops/s
# Warmup Iteration   5: 6.522 ops/s
Iteration   1: 6.560 ops/s
Iteration   2: 6.588 ops/s
Iteration   3: 6.648 ops/s
Iteration   4: 6.635 ops/s
Iteration   5: 6.536 ops/s

# Run progress: 16.67% complete, ETA 00:18:36
# Fork: 3 of 3
# Warmup Iteration   1: 6.485 ops/s
# Warmup Iteration   2: 6.673 ops/s
# Warmup Iteration   3: 6.676 ops/s
# Warmup Iteration   4: 6.668 ops/s
# Warmup Iteration   5: 6.753 ops/s
Iteration   1: 6.717 ops/s
Iteration   2: 6.742 ops/s
Iteration   3: 6.740 ops/s
Iteration   4: 6.773 ops/s
Iteration   5: 6.733 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  6.697 ±(99.9%) 0.089 ops/s [Average]
  (min, avg, max) = (6.536, 6.697, 6.803), stdev = 0.083
  CI (99.9%): [6.608, 6.785] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=7277273239412 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42307:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:44
# Fork: 1 of 3
# Warmup Iteration   1: 9.612 ops/s
# Warmup Iteration   2: 10.043 ops/s
# Warmup Iteration   3: 10.217 ops/s
# Warmup Iteration   4: 10.164 ops/s
# Warmup Iteration   5: 10.181 ops/s
Iteration   1: 10.264 ops/s
Iteration   2: 10.186 ops/s
Iteration   3: 10.265 ops/s
Iteration   4: 10.070 ops/s
Iteration   5: 10.148 ops/s

# Run progress: 33.33% complete, ETA 00:14:52
# Fork: 2 of 3
# Warmup Iteration   1: 9.473 ops/s
# Warmup Iteration   2: 10.135 ops/s
# Warmup Iteration   3: 10.232 ops/s
# Warmup Iteration   4: 10.097 ops/s
# Warmup Iteration   5: 10.236 ops/s
Iteration   1: 10.131 ops/s
Iteration   2: 10.187 ops/s
Iteration   3: 10.094 ops/s
Iteration   4: 10.281 ops/s
Iteration   5: 10.216 ops/s

# Run progress: 41.67% complete, ETA 00:13:00
# Fork: 3 of 3
# Warmup Iteration   1: 9.452 ops/s
# Warmup Iteration   2: 10.225 ops/s
# Warmup Iteration   3: 10.265 ops/s
# Warmup Iteration   4: 10.287 ops/s
# Warmup Iteration   5: 10.243 ops/s
Iteration   1: 10.222 ops/s
Iteration   2: 10.228 ops/s
Iteration   3: 10.223 ops/s
Iteration   4: 10.301 ops/s
Iteration   5: 10.195 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  10.201 ±(99.9%) 0.072 ops/s [Average]
  (min, avg, max) = (10.070, 10.201, 10.301), stdev = 0.067
  CI (99.9%): [10.129, 10.273] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=7277273239412 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42307:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:08
# Fork: 1 of 3
# Warmup Iteration   1: 1.220 ops/s
# Warmup Iteration   2: 1.212 ops/s
# Warmup Iteration   3: 1.240 ops/s
# Warmup Iteration   4: 1.258 ops/s
# Warmup Iteration   5: 1.246 ops/s
Iteration   1: 1.241 ops/s
Iteration   2: 1.247 ops/s
Iteration   3: 1.260 ops/s
Iteration   4: 1.255 ops/s
Iteration   5: 1.257 ops/s

# Run progress: 58.33% complete, ETA 00:09:20
# Fork: 2 of 3
# Warmup Iteration   1: 1.175 ops/s
# Warmup Iteration   2: 1.197 ops/s
# Warmup Iteration   3: 1.214 ops/s
# Warmup Iteration   4: 1.224 ops/s
# Warmup Iteration   5: 0.893 ops/s
Iteration   1: 1.214 ops/s
Iteration   2: 1.216 ops/s
Iteration   3: 1.214 ops/s
Iteration   4: 1.217 ops/s
Iteration   5: 1.226 ops/s

# Run progress: 66.67% complete, ETA 00:07:29
# Fork: 3 of 3
# Warmup Iteration   1: 1.178 ops/s
# Warmup Iteration   2: 1.206 ops/s
# Warmup Iteration   3: 1.219 ops/s
# Warmup Iteration   4: 1.219 ops/s
# Warmup Iteration   5: 1.222 ops/s
Iteration   1: 1.215 ops/s
Iteration   2: 1.216 ops/s
Iteration   3: 1.219 ops/s
Iteration   4: 1.220 ops/s
Iteration   5: 1.221 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.229 ±(99.9%) 0.019 ops/s [Average]
  (min, avg, max) = (1.214, 1.229, 1.260), stdev = 0.017
  CI (99.9%): [1.211, 1.248] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=7277273239412 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=42307:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:38
# Fork: 1 of 3
# Warmup Iteration   1: 0.533 ops/s
# Warmup Iteration   2: 0.581 ops/s
# Warmup Iteration   3: 0.589 ops/s
# Warmup Iteration   4: 0.568 ops/s
# Warmup Iteration   5: 0.583 ops/s
Iteration   1: 0.585 ops/s
Iteration   2: 0.582 ops/s
Iteration   3: 0.590 ops/s
Iteration   4: 0.573 ops/s
Iteration   5: 0.592 ops/s

# Run progress: 83.33% complete, ETA 00:03:46
# Fork: 2 of 3
# Warmup Iteration   1: 0.541 ops/s
# Warmup Iteration   2: 0.583 ops/s
# Warmup Iteration   3: 0.580 ops/s
# Warmup Iteration   4: 0.590 ops/s
# Warmup Iteration   5: 0.588 ops/s
Iteration   1: 0.587 ops/s
Iteration   2: 0.585 ops/s
Iteration   3: 0.595 ops/s
Iteration   4: 0.586 ops/s
Iteration   5: 0.591 ops/s

# Run progress: 91.67% complete, ETA 00:01:54
# Fork: 3 of 3
# Warmup Iteration   1: 0.550 ops/s
# Warmup Iteration   2: 0.598 ops/s
# Warmup Iteration   3: 0.599 ops/s
# Warmup Iteration   4: 0.573 ops/s
# Warmup Iteration   5: 0.603 ops/s
Iteration   1: 0.605 ops/s
Iteration   2: 0.599 ops/s
Iteration   3: 0.603 ops/s
Iteration   4: 0.600 ops/s
Iteration   5: 0.605 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.592 ±(99.9%) 0.010 ops/s [Average]
  (min, avg, max) = (0.573, 0.592, 0.605), stdev = 0.009
  CI (99.9%): [0.582, 0.602] (assumes normal distribution)


# Run complete. Total time: 00:22:54

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   6.697 ± 0.089  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  10.201 ± 0.072  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.229 ± 0.019  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.592 ± 0.010  ops/s

Process finished with exit code 0

May 10 '20 16:05 viluon

I ran PerformanceBenchmark in a similarly "close-to-pristine" environment for https://github.com/SquidDev/Cobalt/pull/38/commits/4f5ed3dea3ce34faa3b3c8dfd9b253d415e8283d.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   7.192 ± 0.074  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  12.171 ± 0.221  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.186 ± 0.005  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.591 ± 0.011  ops/s

Any ideas on what to do about these discrepancies in performance?

PerformanceBenchmark output

/usr/lib/jvm/java-8-openjdk/bin/java -Dvisualvm.id=12498652926297 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=41969:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-openjdk/jre/lib/jfxswt.jar:/home/viluon/Projects/Java/Cobalt/out/test/classes:/home/viluon/Projects/Java/Cobalt/out/test/resources:/home/viluon/Projects/Java/Cobalt/out/production/classes:/home/viluon/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.23/eb242d3261f3795c8bf09818d17c3241191284a0/jmh-core-1.23.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.6.0/b28e078d4e8424de01df02ec92410d225e5d6444/junit-jupiter-params-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.0/f29e6318333d2303ce4965c9819cfad08de7d1e5/junit-jupiter-api-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-library/2.2/cf530c8a0bc993487c64e940ae639bb4a6104dc6/hamcrest-library-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.0/b0a75795cf03841d4f9cc54099557baffc11c727/junit-platform-commons-1.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/2.2/3f2bd07716a31c395e2837254f37f21f0f0ab24b/hamcrest-core-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.2/1820c0968dba3a11a1b30669bb1f01978a91dedc/hamcrest-2.2.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.0/83c9e737f6015d9e00029b9b1d51e952a884b8f9/junit-jupiter-engine-5.6.0.jar:/home/viluon/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.0/a3a6ec96c010875444b3ca31828108093758ec00/junit-platform-engine-1.6.0.jar org.squiddev.cobalt.PerformanceBenchmark
# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=12498652926297 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=41969:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.binarytrees

# Run progress: 0.00% complete, ETA 00:22:00
# Fork: 1 of 3
# Warmup Iteration   1: 6.278 ops/s
# Warmup Iteration   2: 6.993 ops/s
# Warmup Iteration   3: 7.063 ops/s
# Warmup Iteration   4: 7.125 ops/s
# Warmup Iteration   5: 7.141 ops/s
Iteration   1: 7.129 ops/s
Iteration   2: 7.176 ops/s
Iteration   3: 7.139 ops/s
Iteration   4: 7.153 ops/s
Iteration   5: 7.115 ops/s

# Run progress: 8.33% complete, ETA 00:20:30
# Fork: 2 of 3
# Warmup Iteration   1: 6.894 ops/s
# Warmup Iteration   2: 7.220 ops/s
# Warmup Iteration   3: 7.185 ops/s
# Warmup Iteration   4: 7.213 ops/s
# Warmup Iteration   5: 7.260 ops/s
Iteration   1: 7.277 ops/s
Iteration   2: 7.231 ops/s
Iteration   3: 7.187 ops/s
Iteration   4: 7.043 ops/s
Iteration   5: 7.223 ops/s

# Run progress: 16.67% complete, ETA 00:18:36
# Fork: 3 of 3
# Warmup Iteration   1: 6.892 ops/s
# Warmup Iteration   2: 7.149 ops/s
# Warmup Iteration   3: 7.062 ops/s
# Warmup Iteration   4: 7.123 ops/s
# Warmup Iteration   5: 7.253 ops/s
Iteration   1: 7.224 ops/s
Iteration   2: 7.237 ops/s
Iteration   3: 7.256 ops/s
Iteration   4: 7.307 ops/s
Iteration   5: 7.176 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.binarytrees":
  7.192 ±(99.9%) 0.074 ops/s [Average]
  (min, avg, max) = (7.043, 7.192, 7.307), stdev = 0.069
  CI (99.9%): [7.118, 7.266] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=12498652926297 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=41969:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.fannkuch

# Run progress: 25.00% complete, ETA 00:16:44
# Fork: 1 of 3
# Warmup Iteration   1: 11.122 ops/s
# Warmup Iteration   2: 11.534 ops/s
# Warmup Iteration   3: 12.112 ops/s
# Warmup Iteration   4: 12.090 ops/s
# Warmup Iteration   5: 12.258 ops/s
Iteration   1: 12.296 ops/s
Iteration   2: 12.230 ops/s
Iteration   3: 12.302 ops/s
Iteration   4: 12.277 ops/s
Iteration   5: 11.529 ops/s

# Run progress: 33.33% complete, ETA 00:14:51
# Fork: 2 of 3
# Warmup Iteration   1: 11.613 ops/s
# Warmup Iteration   2: 12.265 ops/s
# Warmup Iteration   3: 12.275 ops/s
# Warmup Iteration   4: 12.243 ops/s
# Warmup Iteration   5: 12.287 ops/s
Iteration   1: 12.219 ops/s
Iteration   2: 12.208 ops/s
Iteration   3: 12.187 ops/s
Iteration   4: 12.148 ops/s
Iteration   5: 12.266 ops/s

# Run progress: 41.67% complete, ETA 00:13:00
# Fork: 3 of 3
# Warmup Iteration   1: 11.128 ops/s
# Warmup Iteration   2: 12.127 ops/s
# Warmup Iteration   3: 12.203 ops/s
# Warmup Iteration   4: 12.181 ops/s
# Warmup Iteration   5: 12.187 ops/s
Iteration   1: 12.323 ops/s
Iteration   2: 12.203 ops/s
Iteration   3: 11.872 ops/s
Iteration   4: 12.239 ops/s
Iteration   5: 12.263 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.fannkuch":
  12.171 ±(99.9%) 0.221 ops/s [Average]
  (min, avg, max) = (11.529, 12.171, 12.323), stdev = 0.207
  CI (99.9%): [11.950, 12.392] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=12498652926297 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=41969:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nbody

# Run progress: 50.00% complete, ETA 00:11:08
# Fork: 1 of 3
# Warmup Iteration   1: 1.161 ops/s
# Warmup Iteration   2: 1.178 ops/s
# Warmup Iteration   3: 1.173 ops/s
# Warmup Iteration   4: 1.192 ops/s
# Warmup Iteration   5: 1.184 ops/s
Iteration   1: 1.188 ops/s
Iteration   2: 1.183 ops/s
Iteration   3: 1.192 ops/s
Iteration   4: 1.177 ops/s
Iteration   5: 1.180 ops/s

# Run progress: 58.33% complete, ETA 00:09:19
# Fork: 2 of 3
# Warmup Iteration   1: 1.156 ops/s
# Warmup Iteration   2: 1.179 ops/s
# Warmup Iteration   3: 1.180 ops/s
# Warmup Iteration   4: 1.187 ops/s
# Warmup Iteration   5: 1.182 ops/s
Iteration   1: 1.188 ops/s
Iteration   2: 1.182 ops/s
Iteration   3: 1.184 ops/s
Iteration   4: 1.186 ops/s
Iteration   5: 1.192 ops/s

# Run progress: 66.67% complete, ETA 00:07:29
# Fork: 3 of 3
# Warmup Iteration   1: 1.162 ops/s
# Warmup Iteration   2: 1.181 ops/s
# Warmup Iteration   3: 1.171 ops/s
# Warmup Iteration   4: 1.184 ops/s
# Warmup Iteration   5: 1.183 ops/s
Iteration   1: 1.186 ops/s
Iteration   2: 1.188 ops/s
Iteration   3: 1.187 ops/s
Iteration   4: 1.190 ops/s
Iteration   5: 1.185 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nbody":
  1.186 ±(99.9%) 0.005 ops/s [Average]
  (min, avg, max) = (1.177, 1.186, 1.192), stdev = 0.004
  CI (99.9%): [1.181, 1.190] (assumes normal distribution)


# JMH version: 1.23
# VM version: JDK 1.8.0_242, OpenJDK 64-Bit Server VM, 25.242-b08
# VM invoker: /usr/lib/jvm/java-8-openjdk/jre/bin/java
# VM options: -server -Dvisualvm.id=12498652926297 -javaagent:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/lib/idea_rt.jar=41969:/home/viluon/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-1/201.6668.121/bin -Dfile.encoding=UTF-8 -server -disablesystemassertions
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 12000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.squiddev.cobalt.PerformanceBenchmark.nsieve

# Run progress: 75.00% complete, ETA 00:05:37
# Fork: 1 of 3
# Warmup Iteration   1: 0.556 ops/s
# Warmup Iteration   2: 0.586 ops/s
# Warmup Iteration   3: 0.586 ops/s
# Warmup Iteration   4: 0.595 ops/s
# Warmup Iteration   5: 0.602 ops/s
Iteration   1: 0.604 ops/s
Iteration   2: 0.609 ops/s
Iteration   3: 0.588 ops/s
Iteration   4: 0.605 ops/s
Iteration   5: 0.604 ops/s

# Run progress: 83.33% complete, ETA 00:03:47
# Fork: 2 of 3
# Warmup Iteration   1: 0.546 ops/s
# Warmup Iteration   2: 0.572 ops/s
# Warmup Iteration   3: 0.582 ops/s
# Warmup Iteration   4: 0.561 ops/s
# Warmup Iteration   5: 0.585 ops/s
Iteration   1: 0.585 ops/s
Iteration   2: 0.583 ops/s
Iteration   3: 0.589 ops/s
Iteration   4: 0.574 ops/s
Iteration   5: 0.589 ops/s

# Run progress: 91.67% complete, ETA 00:01:54
# Fork: 3 of 3
# Warmup Iteration   1: 0.561 ops/s
# Warmup Iteration   2: 0.587 ops/s
# Warmup Iteration   3: 0.581 ops/s
# Warmup Iteration   4: 0.595 ops/s
# Warmup Iteration   5: 0.593 ops/s
Iteration   1: 0.593 ops/s
Iteration   2: 0.593 ops/s
Iteration   3: 0.576 ops/s
Iteration   4: 0.595 ops/s
Iteration   5: 0.585 ops/s


Result "org.squiddev.cobalt.PerformanceBenchmark.nsieve":
  0.591 ±(99.9%) 0.011 ops/s [Average]
  (min, avg, max) = (0.574, 0.591, 0.609), stdev = 0.011
  CI (99.9%): [0.580, 0.603] (assumes normal distribution)


# Run complete. Total time: 00:22:54

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                          Mode  Cnt   Score   Error  Units
PerformanceBenchmark.binarytrees  thrpt   15   7.192 ± 0.074  ops/s
PerformanceBenchmark.fannkuch     thrpt   15  12.171 ± 0.221  ops/s
PerformanceBenchmark.nbody        thrpt   15   1.186 ± 0.005  ops/s
PerformanceBenchmark.nsieve       thrpt   15   0.591 ± 0.011  ops/s

Process finished with exit code 0

May 10 '20 17:05 viluon

I've absolutely no clue - I ran the benchmarks using the same setup as the above graph, and had a massive performance decay which is not in line with what's above.

JMH is meant to be pretty good at eliminating this, so I'm really not sure.

May 10 '20 18:05 SquidDev

Cobalt Cobalt copied to clipboard

Replace interpreter with partial evaluation

Approach

Status

Performance comparison

Cobalt
Cobalt copied to clipboard