bddicken
Add Benchmark_Rules.md
There are so many unnecessary issues/PR/discussions, just because of 1 reason. There is no defined benchmarking rules mentioned by author @bddicken .
I suggest him to add BENCHMARK_RULES.md for his project, else there is no point creating issues and PRs.
Maybe,
- Code should use only builtin libraries
- Language version should be latest stable version as on 1st Dec 2024.
- Each benchmark should be simple and test only 1 aspect.
Originally posted by @rsb-23 in https://github.com/bddicken/languages/issues/71#issuecomment-2503021100
As compiling times are not measured, I think boostraping for interpreted languages shouldn't be measured. I mean, only the diference between timestamp before and after the loops should be measured and compared.
As code is run thrice and minimum time is taken, but only 1st run adds to compile time. This behavior is true for python, not sure about others.
With the new runner we only measure the time it takes to run the benchmarked function. So start times and boot strap times and any such time is not included. The runner also makes warmup runs in the same process. See more here:
- #371
More on the point of the OP, the README, and each benchmark README has been updated with attempts to more clarity based on the PRs and issues we've seen.
