Identify and prioritize open perf benchmarks

[smlambert]

We want to select various (open-source) performance benchmark suites/harnesses to run at AdoptOpenJDK (for validating JDK binaries and to make available for easy use by JDK developers).

We can first list any that could be useful for JDK measurement/verification, identify why they are interesting, and then prioritize accordingly. This list can include both full suites and microbenchmarks with links to git repos (and pre-compiled jar/zip files when available).

Once we start adding test targets from different benches, we will also have to ensure that they are suitable for all implementations (not based on implementation specific classes, etc). But we can leave that as a separate exercise.

Larger suites HiBench: https://github.com/Intel-bigdata/HiBench

AcmeAir: https://github.com/blueperf/acmeair-monolithic-java https://acmeair.github.io/acmeair/

DayTrader3: https://github.com/WASdev/sample.daytrader3

DayTrader7: https://github.com/WASdev/sample.daytrader7

Dacapo: https://github.com/dacapobench/dacapobench http://dacapobench.org/

Micro/Macro Bench Harnesses Bumblebench: https://github.com/AdoptOpenJDK/bumblebench

jmh: https://openjdk.java.net/projects/code-tools/jmh/

Also look at https://github.com/Hyperfoil/

[piyush286]

Thanks Shelley for creating this list! Besides the ones that you've already mentioned, here are some more open benchmarks:

Startup time, Footprint and Throughput of OpenLiberty with different apps: https://github.com/OpenLiberty/open-liberty + DT3, DT7, AcmeAir & More

Linkerd: https://github.com/linkerd/linkerd-examples/tree/master/perf-baseline/linkerd1-perf

Databricks: https://github.com/databricks/spark-perf

ES Rally: https://github.com/elastic/rally

Spring Boot Startup Benchmark: https://github.com/dsyer/spring-boot-startup-bench

FrameworkBenchmarks: https://github.com/TechEmpower/FrameworkBenchmarks

Perf Team's Microbenchmarks We've also written a few micro-benchmarks such as for crypto and GPU over the few years so we could think of adding those if you think they might be interesting and useful.

SPEC I guess we're just talking about open benchmarks here but we can also think of running some benchmarks from Standard Performance Evaluation Corporation (SPEC) such as SPECjbb2015 if it's possible to get licenses for them. It would be good to have them as part of AdoptOpenJDK quality assurance and possibly publish official results (https://www.spec.org/jbb2015/results/jbb2015.html) for our releases.

Priority I've already got started on adding Open Liberty with different apps (DT3, DT7 & AcmeAir) since that's the highest priority for us. I'm planning to schedule a meeting soon to discuss about perf testing. FYI @vijaysun-omr

[smlambert]

Thanks @piyush286 ! I have also added a note to the AdoptOpenJDK testing channel to solicit feedback from others (on what open benchmarks they choose, and why)... and looks like we have a nice selection already.

[smlambert]

I also agree that we should consider SPEC benchmarks. We will have to look at the licenses, to understand all of the details.

As I mentioned in our meeting, it could be that we start with the scenario that we add the metadata for running them (as special level), and treat it like JCKs, where folks with licenses could choose to run those targets, moving towards a scenario where we may license & publish results. Ultimately, we will want to look closely at the details, and decide if those benchmarks meet the criteria (open/transparent) and therefore fit into the AQA story or not.

[jerboaa]

https://renaissance.dev/

To quote their website:

Renaissance is a modern, open, and diversified benchmark suite for the JVM, aimed at testing JIT compilers, garbage collectors, profilers, analyzers and other tools.

[smlambert]

https://github.com/Hyperfoil/ also interesting

[smlambert]

Adding https://github.com/spring-projects/spring-petclinic to the list here, so that I can close related: https://github.com/AdoptOpenJDK/openjdk-tests/issues/1501 as we do not need to keep both issues open.

[smlambert]

Instead of tracking it via a separate issue, will close https://github.com/adoptium/aqa-tests/issues/1255 and track databricks and hibench via this issue.