mun
mun copied to clipboard
mun-lang
Add Mun performance regression tests
- [ ] claim this issue (assign yourself or comment below)
- [ ] setup repository on your local machine and make sure all tests pass (
cargo test) - [ ] read our contributing guidelines
- [ ] add a framework (I've been looking at criterion but you're free to use whatever) to benchmark Mun code and add regression tests
- [ ] start a Pull Request. Set description to
closes #58. If this is your first PR, welcome :tada: :smile:
This could be combined with #57
Some initial performance benchmarks have been added here: https://github.com/mun-lang/mun/pull/104
@baszalmstra Happy holidays! I'll take you up on some questions- please bear with me as I get up to speed on the project 😅 Firstly, what are the most important parts of Mun to test for performance regressions? Should we begin with the benchmarks linked here? Secondly, how would you recommend setting up the tests? I was thinking it would be useful to run the tests in CI, but criterion recommends against it.
Ah, I missed the similar issue here, which has some details on CI! To be clear, this issue is about testing the performance of the language runtime, and #57 is about compilation performance?
It appears that iai can get around issues regarding having inconsistent hardware in cloud CI. However it seems like the project isn't maintained. What are your thoughts on using it?
Note: rustc uses rustc-perf, which sadly doesn't provide a crate. It provides a mix of wall-time and hardware-independent metrics like instruction count.
After some thought, I feel that the best course of action is to create some tests (similar to the ones in benchmark.rs) using iai. In another issue, we can create a CI regression testing interface in the same vein as rustc-perf. How does this sound?
@ivyraine that sounds good! We can also run the benchmarks on dedicated hardware from time to time to get consistent results. Alternatively, we could also invest in dedicated hardware.
This issue is about measuring the performance of compiled Mun code indeed. The biggest thing we want to achieve here is to spot regressions more easily.