BenchmarkDotNet icon indicating copy to clipboard operation
BenchmarkDotNet copied to clipboard

Consider adding tests for WASM support

Open adamsitnik opened this issue 3 years ago • 3 comments
trafficstars

WASM support got again broken very recently: https://github.com/dotnet/runtime/issues/78575

We should consider adding automated tests for verifying that WASM just works.

Our Linux CI jobs takes 20+ minutes to execute, so there is plenty of time we can consume (the limit is 1h).

We should mimic what we do for NativeAOT:

  1. Install dependencies on Linux: https://github.com/dotnet/BenchmarkDotNet/blob/ad8e9b26ef22aa77abff2469f7668986e1449127/azure-pipelines.Ubuntu.yml#L17-L22 https://github.com/dotnet/BenchmarkDotNet/blob/ad8e9b26ef22aa77abff2469f7668986e1449127/.github/workflows/build.yaml#L26-L27
  2. Run against hardcoded version to avoid WASM breaking us.

Ideally, we should update the SDK to .NET 7 first: https://github.com/dotnet/BenchmarkDotNet/issues/2080

We don't need to build entire monoVM as described in https://benchmarkdotnet.org/articles/configs/toolchains.html#wasm. We just need to install v8 as described in https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/libraries/testing-wasm.md and follow steps described by @radical here: https://github.com/dotnet/BenchmarkDotNet/issues/1818#issuecomment-1208443917

cc @radical @naricc

adamsitnik avatar Nov 21 '22 13:11 adamsitnik

I think wasm tests here can avoid building the latest dotnet/runtime as you mentioned:

  1. use latest dotnet 7.0
  2. dotnet workload install wasm-tools --skip-manifest-update - this would allow running the tests with relinking, and AOT.
  3. install v8
  4. run some tests!

Essentially, the tests here will provide a basic sanity check that wasm apps can be benchmarked. In addition, the dotnet/performance repo should have similar tests but those will run some microbenchmarks from dotnet/runtime as a sanity check. And that can follow the sdk version being used by dotnet/runtime for testing.

radical avatar Nov 21 '22 19:11 radical

@radical Do we need to rework how the wasm toolchain works? Or do we already support running it against whatever is in the the workload?

naricc avatar Nov 29 '22 14:11 naricc

If you are asking whether running against the workload needs to be implemented, then no, it already works, and dotnet/runtime+dotnet/performance use that.

radical avatar Nov 29 '22 17:11 radical