criterion.rs
criterion.rs copied to clipboard
bheisler
wasm-bindgen support?
Have you considered supporting benchmarks for wasm-bindgen?
rustwasm/wasm-bindgen#828 brings up this issue.
I think the idea would be to reuse some of the wasm-bindgen-test code to run benchmarks in the browser. I'm happy to help with this if you don't have the bandwidth.
CC @chinedufn @fitzgen
A while ago I got criterion compiling on wasm: https://github.com/fitzgen/criterion.rs/tree/wasm
We need some extensions to wasm-bindgen-test for profiling and things though.
Also, we have "add benchmarking support to wasm-bindgen-test" so it would be cool if we used criterion right out the gate!
+cc @alexcrichton
Hey, thanks for the patience. I've been on vacation for a week and a half or so.
Yeah, I'd love to support WASM in Criterion.rs! I have absolutely no idea how to do that, however. Do we even have access to a reasonably precise timer in WASM? I thought browsers limit the precision of timers for security reasons.
I think the most accurate that JS has is performance.now() which isn't really all that accurate (millisecond I think?), but it'd still be neat to have at least a basic form of benchmarking!
You can disable the rounding in Firefox
https://developer.mozilla.org/en-US/docs/Web/API/Performance/now#Reduced_time_precision
In which case we’d have accuracy around 5 microseconds
It looks like Geckodriver can set preferences for the headless browser.
So this could possibly be seamless on the wasm-bindgen-test (or wasm-bindgen-bench) side if we disabled the time rounding automatically (and only let you benchmark in browsers that let us change that setting.)
@bheisler is 5 microseconds small enough - or still too large?
That's probably fine. I assume that if you disable the security feature, browsers are probably just forwarding the call to the same OS time API that Rust libstd (and thus Criterion.rs) uses.
As I say though, I have no experience at all with WASM or any of the tools or requirements of that platform. Pull requests would be welcome, of course.
I think it actually might be more than an order of magnitude less accurate than just directly hitting the OS time API https://stackoverflow.com/a/48212114 so was just making sure that wouldn’t be an issue for the design here (I’m not too familiar with criterion yet)!
Sounds good!
I think the most accurate that JS has is performance.now() which isn't really all that accurate (millisecond I think?), but it'd still be neat to have at least a basic form of benchmarking!
If targeting Node.js, you can also use https://www.npmjs.com/package/microtime which is what Benchmark.js uses when available.
OK, I may have inadvertently mislead you folks.
The timer-resolution issue is likely to be a problem, but it's not the main problem. Criterion.rs is already designed to minimize the effects of imprecise timers. It has to be - many of the things we want to time are on the order of single-digit CPU cycles and no OS has a timer that precise. Having an even-less-precise timer will make the measurements a little bit noisier but that's about it.
The main issue is that I don't know enough about the WASM ecosystem to implement this right now, and I'm not likely to have the time to learn in the near future. I'm not even sure what would have to change in order to support WASM. Even if I had that time, other things (like 0.3.0) are higher priority, at least for me. I'd be happy to help advise and collaborate on pull requests to add this functionality, but that's all I can do for the time being.
Personally, I think trying to adapt criterion might be a bit of an overkill for the JS+WASM ecosystem, since it will involve lot of work to port things like HTML generation, plot generation etc., especially for the -unknown target (as opposed to -emscripten).
For now, I'm using Benchmark.js for my wasm32-unknown-unknown targets with a pretty good success, and it feels like it might be a more promising direction to just wrap it into a crate easily consumable from Rust.
#266 might be relevant here - if all of the HTML generation and plot-rendering stuff were done by a cargo plugin, a hypothetical WASM port would only have to handle measurement, analysis and saving JSON/CSV files to disk, which should be easier.
It would be really nice to be able to compare native and in-browser performance of the same rust lib.
You can compare native vs. in-browser performance using wasi, no?
I don't know if wasi performance is the same for wasm and I don't know if my lib compiles for wasm32-wasi. I'll take further look into it.
WASI is WASM + a systems api. The performance is exactly the same and anything that can compile to wasm can compile to wasi.
You can read more here: https://www.tweag.io/blog/2022-03-03-criterion-rs/#first-class-wasm-support
You can compare native vs. in-browser performance using wasi, no?
The problem is that 1) while there are some polyfills (which might have different performance characteristics than native), WASI is not natively supported in browsers and 2) you can't use wasm-bindgen APIs on WASI target, which limits the benchmarking only to "pure" Rust code.
For example, I'd love to use criterion for benchmarking serde-wasm-bindgen, but currently I have to resort to Benchmark.js on the JS side instead due to those limitations.
Benchmarking in-browser would be a neat feature to have, but it's something that I personally don't need and so I don't have an informed opinion on how it should work. If someone else would like to spearhead that project, that'd be great.
No, as mentioned in the last several comments, that one added WASI support not wasm-bindgen which this issue is about.