BenchmarkDotNet
BenchmarkDotNet copied to clipboard
dotnet
Source Generators based Toolchain
From the .NET Team perspective we quite often hit some build-related issues that require us to change BDN source code and update to latest version. This is especially painful with the WASM toolchain which needs to catch up with a very quickly envolving WASM project build requirements. (cc @naricc @radekdoulik @Lxiamail) It's also a problem for early .NET adopters like ASP.NET Team (cc @halter73 @sebastienros @davidfowl) and projects that use a lot of custom MSBuild settings like ML.NET (@eerhardt).
BDN also lacks the possibility to build the benchmarks on one machine, copy it to a different machine that has no .NET SDK installed and run them there (cc @billwert @drewscoggins @ig-sinicyn).
I'be been thinking about it for a while and I believe that we could solve both problems by implementing a toolchain that takes advantage of C# Source Generators. The idea I have is following:
- Use source generators to generate the boilerplate code (in theory all custom MSBuild setting works OOTB)
- The host process starts new benchmarks by using it's own exe path, but with a different set of arguments that are recognized and translated to "find selected benchmark in the assembly and run it".
- Users should be allowed to specify the runner exe path and args without modifying BDN. Example: V8 for WASM, CoreRun for dotnet/runtime builds.
It would defnitely not become the default toolchain (at least for now) because of the following limitations:
- no F# support (me and @AndreyAkinshin care about F#)
- no ability to run benchmarks for a different runtime (no
--runtimessupport)
I am also not sure what would be the performance of C# Source Generators in case of dotnet/performance Microbenchmarks.csproj that contains 3.5k benchmarks (the boilerplate generated with the existing toolchains in dotnet/performance is more than one million lines of source code and Roslyn fails with OOTM for x86 when we try to run all the benchmarks).
cc @jeffhandley @danmoseley
Hi Adam,
I believe you wanted to have another David Fowler on this thread.
Regards, Dave
On Aug 11, 2021, at 6:11 AM, Adam Sitnik @.***> wrote:
From the .NET Team perspective we quite often hit some build-related issues that require us to change BDN source code and update to latest version. This is especially painful with the WASM toolchain which needs to catch up with a very quickly envolving WASM project build requirements. (cc @naricc @radekdoulik @Lxiamail) It's also a problem for early .NET adopters like ASP.NET Team (cc @halter73 @sebastienros @DavidFowler) and projects that use a lot of custom MSBuild settings like ML.NET @.***).
BDN also lacks the possibility to build the benchmarks on one machine, copy it to a different machine that has no .NET SDK installed and run them there (cc @billwert @DrewScoggins @ig-sinicyn).
I'be been thinking about it for a while and I believe that we could solve both problems by implementing a toolchain that takes advantage of C# Source Generators. The idea I have is following:
Use source generators to generate the boilerplate code (in theory all custom MSBuild setting works OOTB) The host process starts new benchmarks by using it's own exe path, but with a different set of arguments that are recognized and translated to "find selected benchmark in the assembly and run it". Users should be allowed to specify the runner exe path and args without modifying BDN. Example: V8 for WASM, CoreRun for dotnet/runtime builds. It would defnitely not become the default toolchain (at least for now) because of the following limitations:
no F# support (me and @AndreyAkinshin care about F#) no ability to run benchmarks for a different runtime (no --runtimes support) I am also not sure what would be the performance of C# Source Generators in case of dotnet/performance Microbenchmarks.csproj that contains 3.5k benchmarks (the boilerplate generated with the existing toolchains in dotnet/performance is more than one million lines of source code and Roslyn fails with OOTM for x86 when we try to run all the benchmarks).
cc @jeffhandley @danmoseley
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.
Almost all the trouble I've had around wasm has been around SDK changes or changes to the all the build infrastructure around wasm apps (like, how to use the AOT compiler, how to bundle a wasm app, etc). So I am not sure I understand how source generators help?
So I am not sure I understand how source generators help?
Instead of applying the fix in BDN logic that generates project files you would fix the project file in dotnet/performance.
@adamsitnik So I am planning to start work on this soon, and am thinking about the design now. I've never done anything with source generators before and want to make sure I understand the idea.
We want to have a tool chain in BDN that calls into generated code, right? With the generated code eventually produced by clients (like the Microbenchmark harness in dotnet/performance).
I am also not sure what you meant in point 2. above:
The host process starts new benchmarks by using it's own exe path, but with a different set of arguments that are recognized and translated to "find selected benchmark in the assembly and run it.
Is that only for remote runs?
I think you have the wrong David Fowler. I'm not involved in this project.
Regards, Dave
On Tue, Jan 18, 2022 at 2:16 PM Nathan Ricci @.***> wrote:
@adamsitnik https://github.com/adamsitnik So I am planning to start work on this soon, and am thinking about the design now. I've never done anything with source generators before and want to make sure I understand the idea.
We want to have a tool chain in BDN that calls into generated code, right? With the generated code eventually produced by clients (like the Microbenchmark harness in dotnet/performance).
I am also not sure what you meant in point 2. above:
The host process starts new benchmarks by using it's own exe path, but with a different set of arguments that are recognized and translated to "find selected benchmark in the assembly and run it.
Is that only for remote runs?
— Reply to this email directly, view it on GitHub https://github.com/dotnet/BenchmarkDotNet/issues/1770#issuecomment-1015840810, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZDZX3LR3AS3424ORODI6TUWXKDJANCNFSM5B6KRJNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
-- Dave Fowler Partner Solutions Engineering | @.*** | m +1 303 888 5055 Read our Technical blog: blog https://blog.cloudera.com/category/technical/ .cloudera.com/category/technical/ https://blog.cloudera.com/category/technical/
Sorry @DavidFowler (but welcome to our project 😉)
@DavidFowl was intended
@DavidFowler as you probably know, you can hit unsubscribe, either on this page on the right side, or using the link in the email you may have gotten.
Adding @radical to the discussion. He is working on the wasm host app prototype, which might provide some of the functionality you are looking for?
The host process starts new benchmarks by using it's own exe path, but with a different set of arguments that are recognized and translated to "find selected benchmark in the assembly and run it".
I think we should have an in-process option for that, too.
Hey, I recently fought quite a bit with custom toolchain and runtime, and I'm also very supportive of a source generator approach! 😊
@adamsitnik one question I was wondering when also looking around about a related issue like #1403: as I understand it, a source generator approach could avoid entirely to generate separate csprojs (and the complication of replicating the build to a different folder as it is suggested in 1403), is that what you had in mind?
