core_bench icon indicating copy to clipboard operation
core_bench copied to clipboard

First benchmark is slower than the rest

Open mooreryan opened this issue 2 years ago • 0 comments

I've noticed that when running benchmarks, the first one that is specified is slower than the rest, even when the same function with the same inputs is benchmarked repeatedly.

Here is a simple example:

open! Core

let rec fib n = if n < 2 then n else fib (n - 1) + fib (n - 2)

let () =
  let bench name f = Core_bench.Bench.Test.create ~name f in
  Command_unix.run
    (Core_bench.Bench.make_command
       [
         bench "fib 1" (fun () -> fib 10);
         bench "fib 2" (fun () -> fib 10);
         bench "fib 3" (fun () -> fib 10);
       ])

And the result of running time dune exec ./bench/a.exe -- -ascii with a couple different values for fib to see if the behavior changes when shorter or longer running functions are being benched.

  Name    Time/Run   Percentage  
 ------- ---------- ------------ 
  fib 1    30.12ns      100.00%  
  fib 2    18.49ns       61.39%  
  fib 3    18.41ns       61.11%  


  Name    Time/Run   Percentage  
 ------- ---------- ------------ 
  fib 1   317.15ns      100.00%  
  fib 2   216.46ns       68.25%  
  fib 3   215.50ns       67.95%  


  Name    Time/Run   Percentage  
 ------- ---------- ------------ 
  fib 1    35.66us      100.00%  
  fib 2    26.90us       75.44%  
  fib 3    26.97us       75.63%  


  Name    Time/Run   Percentage  
 ------- ---------- ------------ 
  fib 1     4.42ms      100.00%  
  fib 2     3.31ms       74.88%  
  fib 3     3.32ms       75.00%  

Is there something wrong with the way I'm setting up the benchmarks that could be causing this?

mooreryan avatar Jun 12 '22 00:06 mooreryan