gnolang
feat: gnovm benchmarking tool
Contributors' checklist...
- [ x] Added new tests, or not needed, or not feasible
- [ x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
- [ x] Updated the official documentation or not needed
- [ x] No breaking changes were made, or a
BREAKING CHANGE: xxxmessage was included in the description - [ x] Added references to related issues and PRs
- [ x] Provided any useful hints for running manual tests
- [ ] Added new benchmarks to generated graphs, if any. More info here.
We build this tool mainly for the following issues
https://github.com/gnolang/gno/issues/1826 https://github.com/gnolang/gno/issues/1828 https://github.com/gnolang/gno/issues/1281 https://github.com/gnolang/gno/issues/1973
We could also use it in the following cases.
https://github.com/gnolang/gno/issues/1973 https://github.com/gnolang/gno/issues/2222
gnobench benchmarks the time consumed for each VM CPU OpCode and persistent access to the store, including marshalling and unmarshalling of realm objects.
Design consideration
Minimum Overhead and Footprint
- Constant build flags enable benchmarking.
- Encode operations and measurements in binary.
- Dump to a local file in binary.
- No logging, printout, or network access involved.
Accuracy
- Pause the timer for storage access while performing VM opcode benchmarking.
- Measure each OpCode execution in nanoseconds.
- Store access includes the duration for Amino marshalling and unmarshalling.
It is built on top of @deelawn's design and framework with @jaekwon's input. https://github.com/gnolang/gno/pull/2073
Usage
Simple mode
The benchmark only involves the GnoVM and the persistent store. It benchmarks the bare minimum components, and the results are isolated from other components. We use standardize gno contract to perform the benchmarking.
This mode is the best for benchmarking each major release and/or changes in GnoVM.
make opcode
make storage
Production mode
It benchmarks the node in the production environment with minimum overhead. We can only benchmark with standardize the contract but also capture the live usage in production environment. It gives us a complete picture of the node perform.
- Build the production node with benchmarking flags:
go build -tags "benchmarkingstorage benchmarkingops" gno.land/cmd/gnoland
-
Run the node in the production environment. It will dump benchmark data to a benchmark.bin file.
-
call the realm contracts at
gno.land/r/x/benchmark/opcodesandgno.land/r/x/benchmark/storage -
Stop the server after the benchmarking session is complete.
-
Run the following command to convert the binary dump:
gnobench -bin path_to_benchmark_bin
it converts the binary dump to results.csv and results_stats.csv.
Results
The benchmarking results are stored in two files:
- The raw results are saved in results.csv.
| Operation | Elapsed Time | Disk IO Bytes |
|---|---|---|
| OpEval | 40333 | 0 |
| OpPopBlock | 208 | 0 |
| OpHalt | 167 | 0 |
| OpEval | 500 | 0 |
| OpInterfaceType | 458 | 0 |
| OpPopBlock | 166 | 0 |
| OpHalt | 125 | 0 |
| OpInterfaceType | 21125 | 0 |
| OpEval | 541 | 0 |
| OpEval | 209 | 0 |
| OpInterfaceType | 334 | 0 |
- The averages and standard deviations are summarized in results_stats.csv.
| Operation | Avg Time | Avg Size | Time Std Dev | Count |
|---|---|---|---|---|
| OpAdd | 101 | 0 | 45 | 300 |
| OpAddAssign | 309 | 0 | 1620 | 100 |
| OpArrayLit | 242 | 0 | 170 | 700 |
| OpArrayType | 144 | 0 | 100 | 714 |
| OpAssign | 136 | 0 | 95 | 2900 |
| OpBand | 92 | 0 | 30 | 100 |
| OpBandAssign | 127 | 0 | 62 | 100 |
| OpBandn | 97 | 0 | 54 | 100 |
| OpBandnAssign | 125 | 0 | 113 | 100 |
| OpBinary1 | 128 | 0 | 767 | 502 |
| OpBody | 127 | 0 | 145 | 13700 |
Codecov Report
Attention: Patch coverage is 19.37173% with 462 lines in your changes missing coverage. Please review.
:loudspeaker: Thoughts on this report? Let us know!
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
🛠 PR Checks Summary
All Automated Checks passed. ✅
Manual Checks (for Reviewers):
- [ ] IGNORE the bot requirements for this PR (force green CI check)
Read More
🤖 This bot helps streamline PR reviews by verifying automated checks and providing guidance for contributors and reviewers.
✅ Automated Checks (for Contributors):
🟢 Maintainers must be able to edit this pull request (more info)
☑️ Contributor Actions:
- Fix any issues flagged by automated checks.
- Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include
BREAKING CHANGEnotes. - Link related issues/PRs, where applicable.
☑️ Reviewer Actions:
- Complete manual checks for the PR, including the guidelines and additional checks if applicable.
📚 Resources:
Debug
Automated Checks
Maintainers must be able to edit this pull request (more info)
If
🟢 Condition met └── 🟢 The pull request was created from a fork (head branch repo: piux2/gno)Then
🟢 Requirement satisfied └── 🟢 Maintainer can modify this pull requestManual Checks
**IGNORE** the bot requirements for this PR (force green CI check)
If
🟢 Condition met └── 🟢 On every pull requestCan be checked by
- Any user with comment edit permission
@mvertes thank you for the comments!
The pr provides:
- Performance measurement capabilities for opcodes (count, execution time) and persistent storage (count, size). This part is similar to performance counter registers in processors. It would better be called
perf(the VM capability) instead ofbenchops(a particular usage of this capability, and not the only possible one, i.e profiling, gas measurement). See for example linux or freebsd
Thank you for the reference.!
I can see that Linux's perf tool measures software and hardware events in a way similar to how we measure opcode execution and storage access times. However, in our case, we use a set of standardized contracts in benchops/gno to evaluate the execution time of each opcode in the Gno VM runtime. This forms the essence of our benchmarking.
To truly implement performance measurement of the VM, similar to what Linux perf does with software and hardware events, we would need to develop a comprehensive new tool.
Alternatively, we might reserve the name 'Gno Performance and Profiling Tools' for measuring the VM's execution of user contracts in terms of gas consumption, similar to how the Go runtime measures performance for Go code.
An executable called
gnobench, to perform a benchmark on a specific scenario (a sample forum board in gno). This executable should not be located under gnovm, but incontribs/benchmark, with its dependencies:
- the readme, makefile, etc in
contrib/benchmark- the executable itself in
contrib/benchmark/cmd/gnobench- benchmarked gno files in
contrib/benchmark/gno- the vm part still in
gnovm/pkg/perf
I moved benchops/cmd out of /pkg and placed it in gnovm/cmd/benchops since it is still part of the VM.
Regarding the build tags to enable or not the counting, I'm not a big fan of that. It brings a lot of hassle, and it's partly useless because counting is necessary in production anyway. The performance gain vs using a variable instead of a constant (requiring a different executable) to enable the feature in opcode procressing was shown negligible when we introduced the debugger, it should be the same here. A former colleague of mine used to say: "Premature optimisation is the root of all evil" :-)
I totally agree and advocate "Premature optimization is the root of all evil".
All we did was using build flag to set constant flag instead of a variable flag to determine non-production code. Go complier removes entire if block if the condition is a constant value. The constant flag removes the over head 10-18% when we try to capture the accurate value of the operation in nano second with a timer