Shopify
Improve benchmark stability with cpusets, nice, and disabling of hyper threading
This adds to run_benchmarks.rb several additional linux features which can help to make the benchmark results more reliable:
- set scheduler priority (niceness) high
- setup two cpusets, one for benchmarks and one for everything else to move existing processes off of benchmark cpu
- disable hyper threading for benchmark cpus
- move parent process off of cpu that will be used for benchmarks
You can see this improve the results when running the same ruby version twice:
it increases the number of benchmarks that compare at 1.000 on two different computers:
By putting all this logic into a module we can easily reuse it from yjit-metrics.
I think it would make sense to eventually move the existing checks for turbo into this module as well.
For the curious, it looks like it isn't finding any cpuset files in CI, it's only setting the nice value there.
Looks pretty good to me but I will also tag Alan and Kokubun to review since it's very important that we get benchmarking right.
Part of the point of --turbo was to obviate sudo (when we're only interested in stats or testing the behavior), but this seems to add more sudo requirements that are not disabled by --turbo.
Can we rename --turbo to --no-sudo and let it turn off all include Sudo tasks? I'm also fine with making it aliased to --no-benchmark-mode to simplify the implementation (but --no-benchmark-mode itself seems a bit too long to me to type locally).
Thanks for the feedback. I would like to consolidate it a bit more and yeah we can have a flag to disable everything. I'll work on it some more.
Since https://github.com/ruby/ruby-bench/pull/434 we now have a lib directory! So there's precedent for library code in there. I think this would be a really good addition.
Can we rename
--turboto--no-sudo
+1 I also run in this situation and specifically I'm on AMD where --turbo doesn't disable all commands using sudo.
