Support for running benchmarks via JMH Runner?

[hbf]

On the Java side, we can use JMH's Runner to define a main method that you can call. This makes it very easy to:

• Define per-benchmark settings (forks, jvmArgs, etc.) and have these settings close (in the same source file!) to the benchmark
• Run individual benchmarks
• No need to generate a JAR, which would be very useful for benchmark development

I'm not sure whether JMH's Runner works with the new JmhBytecodeGenerator that we use on the Scala side but if so, would it be possible to support Runner's? For example, to allow the user of jmh-gradle-plugin to her/himself write:

object MyBenchmarkRunner {
  def main(args: Array[String]): Unit = {
    new Runner(new OptionsBuilder()
      .include("MyBenchmark")
      .forks(1)
      .jvmArgs("-Xms20m", "-Xmx20m")
      .warmupTime(TimeValue.seconds(3))
      .warmupIterations(3)
      .measurementTime(TimeValue.seconds(10))
      .measurementIterations(3).build).run
  }
}

I didn't get this to work, even after a ./gradlew jmhCompileGeneratedClasses:

Error in IntelliJ when running the above MyBenchmarkRunner.main():

/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA.app… com.myproject.MyBenchmark
objc[4021]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java (0x10da884c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10db524e0). One of the two will be used. Which one is undefined.
Exception in thread "main" No benchmarks to run; check the include/exclude regexps.
	at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:261)
	at org.openjdk.jmh.runner.Runner.run(Runner.java:206)
	at com.myproject.MyBenchmark$.main(MyBenchmark.scala:38)
	at com.myproject.MyBenchmark.main(MyBenchmark.scala)

But on the command line I see:

$ find . -name BenchmarkList
./myproject/build/jmh-generated-classes/META-INF/BenchmarkList

[Barteks2x]

I also need this.

This would allow me to use custom classloader (LaunchWrapper in my case) which would make much easier to benchmark code that uses any kind of instrumentation, without copypasting and modifying all of the used/referenced parts.

[vadimmityanin]

any news regarding this?

[shakeelrao]

+1. It seems strange that the only way to provide configuration is through options in build.gradle.

[nioncode]

I just stumbled upon this, because I was confused that the configuration options in build.gradle seem to not get preserved in the jar generated by jmhJar and was looking for a way to persist the configuration somewhere.

Is there any way to build a jar with the desired configuration so that the jar can be sent to another machine and simply be executed there via java -jar mybenchmark.jar? Currently it seems the only way is to launch the benchmark with the desired options manually like java -jar mybenchmark.jar -wi 5 -bm avgt.