cairo_native
cairo_native copied to clipboard
Use SierraGenerator for testing (part 3)
This PR changes some tests to use Sierra code, rather than Cairo code.
This has the following advantages:
- Reduces test execution time by omitting compilation.
- We ensure that the test is not modified by changes in the compiler
Related to #1152
Checklist
- [x] Linked to Github Issue
Benchmark results Main vs HEAD.
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base dict_insert.cairo (JIT) |
5.251 ± 0.018 | 5.208 | 5.275 | 1.01 ± 0.01 |
base dict_insert.cairo (AOT) |
5.183 ± 0.041 | 5.126 | 5.273 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head dict_insert.cairo (JIT) |
5.285 ± 0.090 | 5.166 | 5.421 | 1.02 ± 0.02 |
head dict_insert.cairo (AOT) |
5.185 ± 0.072 | 5.077 | 5.280 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base dict_snapshot.cairo (JIT) |
5.112 ± 0.033 | 5.052 | 5.154 | 1.02 ± 0.01 |
base dict_snapshot.cairo (AOT) |
5.029 ± 0.027 | 4.983 | 5.069 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head dict_snapshot.cairo (JIT) |
5.122 ± 0.097 | 5.002 | 5.281 | 1.01 ± 0.03 |
head dict_snapshot.cairo (AOT) |
5.057 ± 0.115 | 4.890 | 5.260 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base factorial_2M.cairo (JIT) |
5.491 ± 0.035 | 5.445 | 5.543 | 1.01 ± 0.01 |
base factorial_2M.cairo (AOT) |
5.424 ± 0.020 | 5.387 | 5.454 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head factorial_2M.cairo (JIT) |
5.413 ± 0.106 | 5.285 | 5.604 | 1.00 |
head factorial_2M.cairo (AOT) |
5.538 ± 0.141 | 5.345 | 5.730 | 1.02 ± 0.03 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base fib_2M.cairo (JIT) |
5.034 ± 0.028 | 4.993 | 5.081 | 1.01 ± 0.01 |
base fib_2M.cairo (AOT) |
4.986 ± 0.029 | 4.931 | 5.039 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head fib_2M.cairo (JIT) |
5.036 ± 0.043 | 4.979 | 5.119 | 1.01 ± 0.02 |
head fib_2M.cairo (AOT) |
5.000 ± 0.107 | 4.864 | 5.128 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base linear_search.cairo (JIT) |
5.234 ± 0.042 | 5.203 | 5.333 | 1.02 ± 0.01 |
base linear_search.cairo (AOT) |
5.142 ± 0.038 | 5.074 | 5.204 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head linear_search.cairo (JIT) |
5.231 ± 0.099 | 5.092 | 5.428 | 1.03 ± 0.02 |
head linear_search.cairo (AOT) |
5.096 ± 0.062 | 5.001 | 5.179 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base logistic_map.cairo (JIT) |
5.338 ± 0.043 | 5.256 | 5.409 | 1.03 ± 0.01 |
base logistic_map.cairo (AOT) |
5.196 ± 0.034 | 5.140 | 5.239 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head logistic_map.cairo (JIT) |
5.344 ± 0.133 | 5.169 | 5.563 | 1.05 ± 0.03 |
head logistic_map.cairo (AOT) |
5.101 ± 0.094 | 5.016 | 5.312 | 1.00 |
Codecov Report
Attention: Patch coverage is 99.67267% with 2 lines in your changes missing coverage. Please review.
Project coverage is 80.77%. Comparing base (
0a05bba) to head (c1f0d2d).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/values.rs | 80.00% | 2 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #1127 +/- ##
==========================================
+ Coverage 80.49% 80.77% +0.27%
==========================================
Files 110 110
Lines 29605 29838 +233
==========================================
+ Hits 23832 24101 +269
+ Misses 5773 5737 -36
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
Benchmarking results
Benchmark for program dict_insert
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
11.508 ± 0.040 | 11.468 | 11.596 | 2.28 ± 0.02 |
cairo-native (embedded AOT) |
5.042 ± 0.033 | 4.999 | 5.096 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.135 ± 0.063 | 5.078 | 5.304 | 1.02 ± 0.01 |
Benchmark for program dict_snapshot
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
816.6 ± 8.8 | 802.2 | 828.9 | 1.00 |
cairo-native (embedded AOT) |
4875.0 ± 19.4 | 4854.2 | 4909.8 | 5.97 ± 0.07 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
4993.1 ± 20.5 | 4958.2 | 5023.5 | 6.11 ± 0.07 |
Benchmark for program factorial_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.310 ± 0.036 | 5.271 | 5.398 | 1.00 |
cairo-native (embedded AOT) |
5.333 ± 0.018 | 5.298 | 5.360 | 1.00 ± 0.01 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.375 ± 0.043 | 5.321 | 5.459 | 1.01 ± 0.01 |
Benchmark for program fib_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.106 ± 0.033 | 5.078 | 5.191 | 1.05 ± 0.01 |
cairo-native (embedded AOT) |
4.848 ± 0.036 | 4.802 | 4.916 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
4.874 ± 0.027 | 4.836 | 4.915 | 1.01 ± 0.01 |
Benchmark for program linear_search
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
835.2 ± 5.9 | 829.7 | 844.4 | 1.00 |
cairo-native (embedded AOT) |
4976.1 ± 83.6 | 4895.7 | 5196.2 | 5.96 ± 0.11 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5095.0 ± 44.8 | 5062.7 | 5204.6 | 6.10 ± 0.07 |
Benchmark for program logistic_map
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
542.2 ± 4.3 | 533.6 | 549.9 | 1.00 |
cairo-native (embedded AOT) |
5005.5 ± 41.8 | 4951.7 | 5087.8 | 9.23 ± 0.11 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5198.1 ± 55.2 | 5167.2 | 5352.8 | 9.59 ± 0.13 |
This PR makes testing libfuncs quite complicated and its objective can be achieved by other, simpler, means. So, it has no meaning anymore.