disabling clock frequency scaling and dropping page cache

[nickdesaulniers]

Hi! Cool utility, @nathanchance is using it to benchmark some of our toolchains.

I was wondering if hyperfine does anything related to disabling clock frequency scaling?

On Android, if we don't pin the cores' clocks to fixed frequency, the noise in the benchmark frequently makes the results unreliable.

set -e;

get_cpus () {
  echo -n "$(adb shell 'ls /sys/devices/system/cpu' | grep 'cpu[0-9]')"
}

set_userspace_freq() {
  #echo "setting us $1"
  shell_command="echo userspace > /sys/devices/system/cpu/$1/cpufreq/scaling_governor"
  echo -n $(adb shell $shell_command)
}

get_max_freq() {
  #echo "looking for $1"
  shell_command="cat /sys/devices/system/cpu/$1/cpufreq/cpuinfo_max_freq"
  echo -n "$(adb shell $shell_command)"
}

set_freq() {
  shell_command="echo $2 > /sys/devices/system/cpu/$1/cpufreq/scaling_setspeed"
  echo -n "$(adb shell $shell_command)"
}

verify() {
   shell_command="cat /sys/devices/system/cpu/$1/cpufreq/cpuinfo_cur_freq"
   actual_freq=$(adb shell $shell_command)
   if [ "$actual_freq" == "$2" ]; then
     echo "good"
   else
     echo "bad"
   fi
}

echo "finding number of cpus"
for cpu in $(get_cpus)
do
    #echo "$cpu"
    echo "setting cpu frequency scaling governor to userspace control"
    set_userspace_freq $cpu
    echo "finding max freqency of $cpu"
    freq="$(get_max_freq $cpu)"
    echo "setting $cpu to $freq"
    set_freq $cpu $freq
    verify $cpu $freq
done

Is what I usually do; I'm not sure if the same sysfs nodes are exposed on x86.

Another thing I do when benchmarking is clear the page cache rather than do warmup runs:

echo 3 > /proc/sys/vm/drop_caches

I couldn't quickly tell if these are done by hyperfine, but maybe they might help?

[nathanchance]

Dropping the page cache can be done with the --prepare command: https://github.com/sharkdp/hyperfine#warmup-runs-and-preparation-commands

Pinning the scaling freq appears more difficult; for example, on a server with an AMD EPYC 7401P, I don't see any clear way to do it via sysfs. This is a good idea to look into for the LLVM benchmarks though.

[sharkdp]

Hi! Cool utility, @nathanchance is using it to benchmark some of our toolchains.

Thank you very much for the feedback.

I was wondering if hyperfine does anything related to disabling clock frequency scaling?

No, it does not. But I think it's definitely worth to think about adding support in some sense (showing warnings when scaling is activated, providing an easy way to activate it, etc.).

I'm not really familiar with the details of clock frequency scaling and how it can be deactivated, but I know that google-benchmark shows this warning when CPU frequency scaling is activated:

***WARNING*** CPU scaling is enabled, the benchmark real time
measurements may be noisy and will incur extra overhead.

(see https://github.com/google/benchmark#disabling-cpu-frequency-scaling)

They suggest to use

sudo cpupower frequency-set --governor performance

to disable it. cpupower seems to be a Linux-specific tool.

Dropping the page cache can be done with the --prepare command: https://github.com/sharkdp/hyperfine#warmup-runs-and-preparation-commands

Exactly. --prepare runs a specific command before each timing run. For cpupower, we wouldn't really need --prepare because we could just globally deactivate it and re-enable it after the benchmark (if necessary):

sudo cpupower frequency-set --governor performance
hyperfine …
sudo cpupower frequency-set --governor powersave

It looks like cpupower frequency-info can be used to query the current state.

So far, I haven't found an example where I can measure a significant difference between the two modes, but I'd be interested to hear your feedback.

[nickdesaulniers]

So far, I haven't found an example where I can measure a significant difference between the two modes, but I'd be interested to hear your feedback.

This may be more pronounced on mobile.

[sharkdp]

Coming back to this...

google benchmark checks if CPU frequency scaling is enabled by reading the contents of

/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

(see https://github.com/google/benchmark/blob/367119482ff4abc3d73e4a109b410090fc281337/src/sysinfo.cc#L219-L227)

We could easily do the same in hyperfine. However, I'm not sure if we generally want to show a warning if frequency scaling is enabled. Personally, I mostly (but not exclusively) use hyperfine for "IO heavy" benchmarks where I am not too concerned about small variations in CPU speed (which are hard to measure even for CPU-heavy programs, see above). Showing "you might want to disable the CPU frequency scaling" in all cases might be a bit too much.

Automatically disabling frequency scaling is not really possibly as it requires sudo privileges.

Suggestions?

[nickdesaulniers]

I mostly (but not exclusively) use hyperfine for "IO heavy" benchmarks where I am not too concerned about small variations in CPU speed

But other users of hyperfine may not.

which are hard to measure even for CPU-heavy programs, see above

That has not been my experience, but YMMV.

Automatically disabling frequency scaling is not really possibly as it requires sudo privileges.

What does prior art (google benchmark) do?

[sharkdp]

which are hard to measure even for CPU-heavy programs, see above

That has not been my experience, but YMMV.

It's hard for me to implement this feature if I have no example program to test this.

Automatically disabling frequency scaling is not really possibly as it requires sudo privileges.

What does prior art (google benchmark) do?

I have described this above. It just prints a warning.