cairo_native
cairo_native copied to clipboard
Use SierraGenerator for testing (part 2)
This PR changes some tests to use Sierra code, rather than Cairo code.
This has the following advantages:
- Reduces test execution time by omitting compilation.
- We ensure that the test is not modified by changes in the compiler
Related to #1152
Checklist
- [x] Linked to Github Issue
Benchmark results Main vs HEAD.
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base dict_insert.cairo (JIT) |
5.066 ± 0.040 | 5.014 | 5.139 | 1.01 ± 0.01 |
base dict_insert.cairo (AOT) |
5.026 ± 0.043 | 4.970 | 5.115 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head dict_insert.cairo (JIT) |
5.246 ± 0.024 | 5.208 | 5.275 | 1.01 ± 0.01 |
head dict_insert.cairo (AOT) |
5.183 ± 0.027 | 5.141 | 5.242 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base dict_snapshot.cairo (JIT) |
5.071 ± 0.027 | 5.033 | 5.120 | 1.02 ± 0.01 |
base dict_snapshot.cairo (AOT) |
4.948 ± 0.059 | 4.844 | 5.028 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head dict_snapshot.cairo (JIT) |
5.143 ± 0.046 | 5.072 | 5.220 | 1.03 ± 0.02 |
head dict_snapshot.cairo (AOT) |
4.981 ± 0.059 | 4.883 | 5.078 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base factorial_2M.cairo (JIT) |
5.458 ± 0.057 | 5.322 | 5.520 | 1.03 ± 0.01 |
base factorial_2M.cairo (AOT) |
5.311 ± 0.021 | 5.276 | 5.337 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head factorial_2M.cairo (JIT) |
5.463 ± 0.060 | 5.376 | 5.548 | 1.01 ± 0.01 |
head factorial_2M.cairo (AOT) |
5.407 ± 0.047 | 5.321 | 5.487 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base fib_2M.cairo (JIT) |
4.929 ± 0.039 | 4.878 | 4.988 | 1.00 ± 0.01 |
base fib_2M.cairo (AOT) |
4.914 ± 0.053 | 4.843 | 5.006 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head fib_2M.cairo (JIT) |
4.936 ± 0.046 | 4.870 | 5.007 | 1.00 ± 0.01 |
head fib_2M.cairo (AOT) |
4.935 ± 0.055 | 4.862 | 5.014 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base linear_search.cairo (JIT) |
5.178 ± 0.033 | 5.136 | 5.216 | 1.04 ± 0.01 |
base linear_search.cairo (AOT) |
4.993 ± 0.036 | 4.941 | 5.040 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head linear_search.cairo (JIT) |
5.124 ± 0.082 | 5.005 | 5.228 | 1.04 ± 0.02 |
head linear_search.cairo (AOT) |
4.929 ± 0.047 | 4.861 | 5.019 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base logistic_map.cairo (JIT) |
5.296 ± 0.074 | 5.202 | 5.424 | 1.04 ± 0.02 |
base logistic_map.cairo (AOT) |
5.098 ± 0.042 | 5.049 | 5.181 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head logistic_map.cairo (JIT) |
5.178 ± 0.040 | 5.124 | 5.254 | 1.04 ± 0.01 |
head logistic_map.cairo (AOT) |
4.966 ± 0.026 | 4.931 | 5.004 | 1.00 |
Codecov Report
Attention: Patch coverage is 99.10448% with 3 lines in your changes missing coverage. Please review.
Project coverage is 80.72%. Comparing base (
0a05bba) to head (b746727).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/values.rs | 70.00% | 3 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #1126 +/- ##
==========================================
+ Coverage 80.49% 80.72% +0.22%
==========================================
Files 110 110
Lines 29605 29799 +194
==========================================
+ Hits 23832 24054 +222
+ Misses 5773 5745 -28
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
Benchmarking results
Benchmark for program dict_insert
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
11.445 ± 0.054 | 11.384 | 11.571 | 2.33 ± 0.02 |
cairo-native (embedded AOT) |
4.914 ± 0.037 | 4.875 | 4.992 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.008 ± 0.034 | 4.951 | 5.050 | 1.02 ± 0.01 |
Benchmark for program dict_snapshot
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
795.6 ± 5.8 | 787.3 | 806.1 | 1.00 |
cairo-native (embedded AOT) |
4796.1 ± 33.1 | 4754.9 | 4844.5 | 6.03 ± 0.06 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
4936.8 ± 31.1 | 4885.5 | 5005.6 | 6.21 ± 0.06 |
Benchmark for program factorial_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.301 ± 0.013 | 5.283 | 5.318 | 1.01 ± 0.01 |
cairo-native (embedded AOT) |
5.231 ± 0.026 | 5.190 | 5.260 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.284 ± 0.023 | 5.242 | 5.320 | 1.01 ± 0.01 |
Benchmark for program fib_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.104 ± 0.017 | 5.080 | 5.133 | 1.07 ± 0.01 |
cairo-native (embedded AOT) |
4.775 ± 0.028 | 4.730 | 4.824 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
4.810 ± 0.027 | 4.771 | 4.855 | 1.01 ± 0.01 |
Benchmark for program linear_search
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
819.6 ± 5.8 | 813.4 | 830.3 | 1.00 |
cairo-native (embedded AOT) |
4869.4 ± 30.5 | 4830.8 | 4909.8 | 5.94 ± 0.06 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5001.6 ± 20.7 | 4969.2 | 5039.7 | 6.10 ± 0.05 |
Benchmark for program logistic_map
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
535.5 ± 3.8 | 530.5 | 542.4 | 1.00 |
cairo-native (embedded AOT) |
4933.3 ± 29.6 | 4872.7 | 4972.5 | 9.21 ± 0.09 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5078.2 ± 23.7 | 5034.6 | 5107.4 | 9.48 ± 0.08 |
This PR makes testing libfuncs quite complicated and its objective can be achieved by other, simpler, means. So, it has no meaning anymore.