Use Same JVMs for Everyone

[davecom]

Thank you for this fun and interesting competition and maintaining it so well so everyone can enjoy it. I do have one suggestion:

I think the spirit of the competition should be more around the code, not who can figure out the best JVM to use for their code. So I would suggest two options:

• Use a standard (perhaps best in class for this type of application) JVM for everyone
• Run all submissions on a pre-defined set of JVMs and show the times next to each other (interesting to see, but expensive in terms of time to run)

Comparing one submission running on one JVM to another submission on another JVM is kind of like comparing apples to oranges if they radically differ in terms of execution model (Graal AOT vs HotSpot JIT for example). You have strictly limited the language here (Java, not even other JVM languages), the dependencies (only standard libraries), why not also strictly limit the JVM so it is a level playing-field?

[AlexanderYastrebov]

See also discussion https://github.com/gunnarmorling/1brc/discussions/67 related to Go implementation where we get varying results on different platforms.

[gunnarmorling]

The official results are all obtained by running any entries on one on the same machine, so they generally are comparable (see README for details). It uses dedicated cores (good) but it still is a VM, so no idea whatever other impact their could be. That said, I've observed rather stable results for multiple runs of specific contenders. Note the instance class (CCX33 on Hetzner Cloud) itself is not guaranteed for obtaining stable results. @ebarlas sees much better values on his instance of the same class. That's why I am doing all official runs on that one specific machine. We may do a re-evaluation of the top N contenders at the end in some more controlled environment (if we find one), if results are super close. Needless to say that I can't stop folks sharing numbers from their own machines etc. As you say, they are inherently not comparable.

[davecom]

Sorry I should have been more clear. I meant the same JVM. I will update my issue.

[gunnarmorling]

Ah, got it. But that is intentional actually. Choice of JVM is to allow for different GCs, preview APIs, etc.

[davecom]

I mean you could make the argument that allowing different JVM languages would also be interesting in terms of exploring different programmatic paradigms/memory models/standard libraries. It seems like an arbitrary place to draw the line on one but not the other. But it's your contest, and it's fun either way.

[gunnarmorling]

I agree, there is no exact science to drawing that line. I decided to rule out other JVM languages, as a) I expected them not to make a substantial difference in performance (I don't think for instance they could differ from the Java memory model, nor am I aware of any which would bring potential key differentiators like their own GC or say a vector API, but I might be wrong) and b) I wanted to keep things somewhat simple just in terms of operating the challenge. Which I think was a good call, last few days have been crazy already, as-is :)

In contrast, allowing for different distributions felt like an interesting angle to me, as they bring interesting new capabilities (e.g. Graal JIT, GCs, new APIs from EA builds) which I wanted to shed a light on. Handling is easy enough too via sdkman. With the chosen JDK shown in the scoreboard I think that's all fair. For instance one key take away for me from all this is that running on graal(ce) often may be a quick win, essentially for free.

Finally, other languages (JVM or not) are very much welcomed in the Show & Tell, it's amazing to see what folks come up with. In that light, I am gonna close this one as "Won't do". Thanks a lot for bringing this up (and participating in the challenge!) nevertheless.