Simple way to run benchmark with a ref of a state

[jpochyla]

I understand it's possible to use the matrix + generators for this, but something akin to this simple Criterion code is currently missing, I think:

fn benchmark(c: &mut Criterion) {
    let mut state = compute_heavy_state();
    c.bench_function("foo", |b| {
        b.iter(|| bar(&mut state))
    });
}

[bazhenov]

Now as I'm thinking about it, it seems easy to implement function similar to benchmark_fn(), that will pass state by reference:

fn bench() -> imp IntoBenchmark {
  benchmark_fn_state(compute_heavy_state(), |state| {
    // do work
  })
}

[FilipAndersson245]

Any update on this issue? I have a large mutatable state that I would rather clone and insert to the benchmark, rather than calculate it during benchmarking.

[bazhenov]

Yes, there is new API in dev branch that is very similar to criterion API that is solves the issue. README.md in dev branch has some simple examples. Please let me know if you have any issues with it.

[FilipAndersson245]

I was looking over the code on dev and as I understand its working it would still require me to clone inside the benchmarking function cloneing the eventual heavy state, I think how you can do it in a separate setup function in Criterion is decently clean.

fn bench_load_standard_lib(c: &mut Criterion) {
    c.bench_function("load_standard_lib", |b| {
        let engine = setup_engine();
        b.iter_batched(
            || engine.clone(),
            |mut engine| {
                load_standard_library(&mut engine).unwrap();
            },
            criterion::BatchSize::SmallInput,
        );
    });
}

Doing the equivalent here would still require the cloning to be done inside the b.iter() function? like this?

fn parse_benchmarks() -> impl IntoBenchmarks {
    [benchmark_fn("record", |b| {
        let mut compiler = Compiler::new();

        let content = include_bytes!("record.nu");
        compiler.add_file(&"record.nu", content);

        let parser = Parser::new(compiler, 0);
        b.iter(move || parser.clone().parse())
    })]
}

(not the same, as tango is broken when I tried to migrate the first example away from criterion.)

[bazhenov]

You don't have to clone the state. You can clone the Rc ref. There is a relevant example in source code:

https://github.com/bazhenov/tango/blob/51f4e8414e4aabefda987baa2f26fa8a7c72110f/examples/benches/test_funcs.rs#L9-L20

The reason behind it that tango can rotate the state at a given amount of samples (see: MeasurementSettings::samples_per_haystack). This way you can control effects of CPU-caching behavior.

[FilipAndersson245]

But if I need a mutatable state i cannot use that right?

[bazhenov]

Yes, mutable state is not allowed at the moment. This makes benchmarks less reproducible, because each iteration is not independent in case of mutable state.

[FilipAndersson245]

Okay, We probably need some way to handle mutatable inputs, as most of our API is designed around it. I think as how criterion does it is clean, having a setup function that is ran in between?, but i guess it also makes iterating the bench a lot more noisy, even if that callbacks only usage is to clone the value.

edit: I think maybe if we can run the setup function I times, and store the result in a list to be used for each iteration would at least mitigate the overhead of switching between setup and iterating.

[FilipAndersson245]

Currently when testing out I need to do something like this to get access to the state, that has to be reset between iterations.

fn load_standard_lib() -> impl IntoBenchmarks {
    let engine = setup_engine();
    [benchmark_fn("load_standard_lib", move |b| {
        let engine = engine.clone();
        b.iter(move || {
            let mut engine = engine.clone();
            load_standard_library(&mut engine)
        })
    })]
}

Think it would be very nice with an API for setup.

[bazhenov]

Just to make sure I understand you correctly. You want some nice API to be able to clone mutable state when defining benchmark function. Something along the lines of

fn load_standard_lib() -> impl IntoBenchmarks {
    let engine = setup_engine();
    [benchmark_fn("load_standard_lib", move |b| {
        b.iter_with_state(|| engine.clone(), |e| {
          load_standard_library(&mut e)
        })
    })]
}

[FilipAndersson245]

Yes, that would be very simular to how criterion does, as we have a large mut state and prefer not to have the cloning be part of the measurment.