Support for wasm32-wasi

[bkolobara]

I'm trying to get criterion running on Wasmtime and Wasmer after compiling the application to target=wasm32-wasi.

There is already an issue about getting it running in the browser with wasm-bindgen, but I think that it would be actually more beneficial to get WASI support first. Especially because the WASI specification exposes a higher precision clock and there is currently no alternative to measure performance inside these Wasm runtimes.

I ran into an issue with the plotters dependency and this is currently my main blocker.

[bheisler]

Hello! Thanks for trying Criterion.rs and thanks for the suggestion.

Benchmarking in WASM is not supported by Criterion.rs. I don't even know what would be required to support such. If you really want to blaze that trail yourself and the changes needed aren't too onerous to maintain, I would be open to pull requests though.

I am very skeptical that Criterion.rs can give you reasonable measurements though. JIT compilers add a whole extra level of unreliability to benchmarking results that Criterion.rs' analysis can't really do anything to mitigate. They sometimes make different optimization decisions each time you re-launch the VM.

[bkolobara]

Thanks @bheisler for the fast response and your work on criterion.rs!

I actually managed to hack something together using this PR as the plotters dependency. I just needed to comment out some code in criterion, because it's not compatible with the latest plotters="0.3.0".

It still fails with the following stacktrace:

Gnuplot not found, using plotters backend
Benchmarking Fibonacci/Recursive
Benchmarking Fibonacci/Recursive: Warming up for 3.0000 s
Benchmarking Fibonacci/Recursive: Collecting 100 samples in estimated 5.1665 s (106k iterations)
Benchmarking Fibonacci/Recursive: Analyzing
thread 'main' panicked at 'The global thread pool has not been initialized.: ThreadPoolBuildError { kind: IOError(Custom { kind: Other, error: "operation not supported on this platform" }) }', /Users/bkolobara/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:171:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: failed to run `/Users/bkolobara/Development/lunatic-rust-lib/target/wasm32-wasi/release/deps/lunatic_benchmark-7c4ef4aa98c6dbf1.wasm`
│   1: WASI execution failed
│   2: failed to run WASI `_start` function
│   3: RuntimeError: unreachable
           at __rust_start_panic (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2758]:0x1d7009)
           at rust_panic (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2738]:0x1d66a3)
           at std::panicking::rust_panic_with_hook::h7412819944345424 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2731]:0x1d6319)
           at std::panicking::begin_panic_handler::{{closure}}::h4738c0049ce98322 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2709]:0x1d5795)
           at std::sys_common::backtrace::__rust_end_short_backtrace::h8bb6e3f06234953f (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2708]:0x1d56d6)
           at rust_begin_unwind (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2730]:0x1d611f)
           at core::panicking::panic_fmt::h904ce09f3cb14707 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2881]:0x1e02c4)
           at core::option::expect_none_failed::ha29c9c4296a18e58 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2909]:0x1e50ab)
           at rayon_core::registry::global_registry::hbc2b8d2944538eeb (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[1290]:0x11676a)
           at rayon_core::current_num_threads::h02c70f12917ff759 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[1305]:0x117b25)
           at rayon::iter::plumbing::bridge::h08b28b0882d97241 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[945]:0xe1126)
           at criterion::analysis::estimates::h80051e0ee26a838f (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[632]:0xa2264)
           at criterion::analysis::common::h35334743f9bc3776 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[242]:0x1ed6a)
           at <criterion::benchmark::ParameterizedBenchmark<T,M> as criterion::benchmark::BenchmarkDefinition<M>>::run::h2d7bc7b37c6213f4 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[78]:0xab1d)
           at criterion::Criterion<M>::bench_functions::hc008b5c8685b6941 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[184]:0x12439)
           at lunatic_benchmark::main::h0572fac7ac131b49 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[145]:0xf39e)
           at std::sys_common::backtrace::__rust_begin_short_backtrace::h850e39496a8b8d31 (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[63]:0x9743)
           at std::rt::lang_start::{{closure}}::h0a28dbf7a1278ecc (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[161]:0x10090)
           at std::rt::lang_start_internal::h0e1571f3e9f07dad (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[2741]:0x1d67e6)
           at __original_main (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[146]:0xf44d)
           at _start (lunatic_benchmark-7c4ef4aa98c6dbf1.wasm[18]:0x18cc)
╰─> 4: unreachable
error: bench failed

The problem is that wasm doesn't have threading support yet. I'm not sure if you can configure cirterion to run in a single threaded environment?

But apart from this everything else looks good. I would say that the only change needed is updating plotters and adding single threaded support. I would need to first get a bit more familiar with the codebase, but if you think that this changes are ok I could submit a PR eventually.

Regarding the measurement precision, there are many Wasm runtimes and compilers out. Some, like LLVM and Cranelift are able to pre-compile your .wasm file to an executable so you avoid the uncertainty of JIT. This would make criterion a great tool to compare native vs wasm code and the different compilers.

[bheisler]

Currently no, it's not possible to run without thread support. You would need to implement alternate versions of the functions that currently rely on Rayon, and then tag them with the appropriate #[cfg(...)] attributes so they're compiled instead of the parallel versions on WASM. Possibly you could also make the Rayon dependency itself optional based on the target triple? I don't remember if that's configurable through Cargo.toml.

[gilescope]

@bkolobara feel free to share your fork. At the moment there's no benchmarking for stable wasm, would be great to have something work.

[lemmih]

A basic version of this is now operational. I'd like to see it included in the upcoming 0.4 release.

$ cargo wasi bench --no-default-features -- Fibonacci/Recursive --discard-baseline --noplot
    Finished bench [optimized] target(s) in 0.02s
     Running unittests (target/wasm32-wasi/release/deps/bench_main-8d0cd5847ebd9ccb.wasm)
     Running `/home/lemmih/coding/criterion.rs/target/wasm32-wasi/release/deps/bench_main-8d0cd5847ebd9ccb.wasm Fibonacci/Recursive --discard-baseline --noplot --bench`
Benchmarking Fibonacci/Recursive
Benchmarking Fibonacci/Recursive: Warming up for 3.0000 s
Benchmarking Fibonacci/Recursive: Collecting 100 samples in estimated 5.0306 s (162k iterations)
Benchmarking Fibonacci/Recursive: Analyzing
Fibonacci/Recursive     time:   [30.494 us 30.645 us 30.826 us]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe

It works because rayon and plotters are optional dependencies.

[Mastermindaxe]

Hey everyone! Are there any updates on this?

[bkolobara]

We use this branch for lunatic apps compiled to wasm32-wasi and it works for us:

[dev-dependencies]
# WASI branch
criterion = { git = "https://github.com/bheisler/criterion.rs", branch = "version-0.4", default-features = false  }

[lemmih]

Hey everyone! Are there any updates on this?

WASM support will be part of the upcoming 0.4 release.