Suggestion: Use hyperfine instead of time to perform timing measurements

[christianhujer]

Suggestion: Use hyperfine instead of time for performing timing measurements. It can automatically perform multiple runs and print statistical timing data on those runs.

Maybe I'll find time on the weekend to provide a PR for this.

[hundredwatt]

I made a script using hyperfine that also calculates the result how this challenge prescribes:

TEMP_FILE=$(mktemp)
OPTS="--warmup 2 --runs 5 --export-json $TEMP_FILE"
JAVA_OPTS=""

hyperfine $OPTS \
        'java $JAVA_OPTS --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_hundredwatt' \
        'java $JAVA_OPTS --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_<COMPARE TO FORK>'

# The slowest and the fastest runs are discarded
# The mean value of the remaining three runs is the result for that contender
jq -r '.results[] | .command, ((.times | sort) | .[1:-1] | add / length)' $TEMP_FILE

[gunnarmorling]

Oh, very cool! Could you PR this script? I probably shouldn't change the way of evaluating in the middle of the challenge, but it's good to have for future reference and ad-hoc testing. Thx!

[hundredwatt]

Sure! Here's the PR: #182

[gunnarmorling]

Awesome, thanks. That's a great start. We'd have to factor out though the call to sdk used by some of the contenders to enable a specific JDK distro, as that adds a non-insignificant overhead. Is there some sort of pre-run hook one could use for that? Similar for the additional_build_steps script needed for the native image entries.

[hundredwatt]

Yes, hyperfine supports individual prepare commands, so we could do:

hyperfine $OPTS \
  --prepare './prepare_<fork1>.sh' \
  './calculate_average_<fork1>.sh' \
  --prepare './prepare_<fork2>.sh' \
  './calculate_average_<fork2>.sh'

But we still need to solve the sdkman issue (https://twitter.com/jasonnochlin/status/1744399656266629365)

[hundredwatt]

Resolved on Twitter (https://twitter.com/gunnarmorling/status/1744413791272468607):

source ./prepare_<fork>.sh # jdk use... if present
additional_build_steps_<fork>.sh # if present
hyperfine 'calculate_average_<fork>.sh' # sets JAVA_OPTS and runs

[gunnarmorling]

Done. Thx a lot, @hundredwatt, great work! Makes the eval process much, much smoother now!

[hundredwatt]

My less than 10 line POC above has turned into a 300 line behemoth 😂

I hope I made managing this contest a little less tedious for you @gunnarmorling! Thanks again for creating it!