cairo_native
cairo_native copied to clipboard
Use SierraGenerator for testing (part 5)
This PR changes some tests to use Sierra code, rather than Cairo code.
This has the following advantages:
- Reduces test execution time by omitting compilation.
- We ensure that the test is not modified by changes in the compiler
Related to #1152
Checklist
- [x] Linked to Github Issue
Benchmark results Main vs HEAD.
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base dict_insert.cairo (JIT) |
5.128 ± 0.068 | 5.048 | 5.287 | 1.02 ± 0.01 |
base dict_insert.cairo (AOT) |
5.036 ± 0.018 | 5.005 | 5.069 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head dict_insert.cairo (JIT) |
5.100 ± 0.037 | 5.062 | 5.191 | 1.02 ± 0.01 |
head dict_insert.cairo (AOT) |
4.980 ± 0.028 | 4.936 | 5.028 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base dict_snapshot.cairo (JIT) |
5.012 ± 0.029 | 4.975 | 5.067 | 1.02 ± 0.01 |
base dict_snapshot.cairo (AOT) |
4.917 ± 0.034 | 4.870 | 4.972 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head dict_snapshot.cairo (JIT) |
5.008 ± 0.041 | 4.957 | 5.092 | 1.02 ± 0.02 |
head dict_snapshot.cairo (AOT) |
4.919 ± 0.074 | 4.842 | 5.062 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base factorial_2M.cairo (JIT) |
5.378 ± 0.031 | 5.318 | 5.412 | 1.01 ± 0.01 |
base factorial_2M.cairo (AOT) |
5.330 ± 0.034 | 5.293 | 5.392 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head factorial_2M.cairo (JIT) |
5.379 ± 0.025 | 5.340 | 5.417 | 1.01 ± 0.01 |
head factorial_2M.cairo (AOT) |
5.313 ± 0.045 | 5.253 | 5.381 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base fib_2M.cairo (JIT) |
4.924 ± 0.049 | 4.822 | 4.983 | 1.02 ± 0.01 |
base fib_2M.cairo (AOT) |
4.841 ± 0.036 | 4.805 | 4.917 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head fib_2M.cairo (JIT) |
4.883 ± 0.030 | 4.824 | 4.928 | 1.01 ± 0.01 |
head fib_2M.cairo (AOT) |
4.831 ± 0.036 | 4.762 | 4.888 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base linear_search.cairo (JIT) |
5.163 ± 0.048 | 5.084 | 5.236 | 1.04 ± 0.02 |
base linear_search.cairo (AOT) |
4.984 ± 0.070 | 4.890 | 5.132 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head linear_search.cairo (JIT) |
5.044 ± 0.012 | 5.022 | 5.062 | 1.03 ± 0.01 |
head linear_search.cairo (AOT) |
4.910 ± 0.021 | 4.876 | 4.936 | 1.00 |
Base
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
base logistic_map.cairo (JIT) |
5.206 ± 0.042 | 5.154 | 5.280 | 1.03 ± 0.01 |
base logistic_map.cairo (AOT) |
5.044 ± 0.027 | 5.013 | 5.109 | 1.00 |
Head
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
head logistic_map.cairo (JIT) |
5.147 ± 0.030 | 5.107 | 5.207 | 1.03 ± 0.01 |
head logistic_map.cairo (AOT) |
4.985 ± 0.018 | 4.961 | 5.013 | 1.00 |
Codecov Report
Attention: Patch coverage is 95.00000% with 15 lines in your changes missing coverage. Please review.
Project coverage is 80.64%. Comparing base (
0a05bba) to head (8ee859c).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/values.rs | 88.78% | 12 Missing :warning: |
| src/libfuncs/felt252_dict_entry.rs | 96.66% | 2 Missing :warning: |
| src/libfuncs/felt252.rs | 97.36% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #1129 +/- ##
==========================================
+ Coverage 80.49% 80.64% +0.14%
==========================================
Files 110 110
Lines 29605 29776 +171
==========================================
+ Hits 23832 24014 +182
+ Misses 5773 5762 -11
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
Benchmarking results
Benchmark for program dict_insert
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
11.792 ± 0.080 | 11.655 | 11.892 | 2.33 ± 0.04 |
cairo-native (embedded AOT) |
5.070 ± 0.080 | 4.977 | 5.246 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.144 ± 0.045 | 5.077 | 5.223 | 1.01 ± 0.02 |
Benchmark for program dict_snapshot
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
883.1 ± 15.2 | 859.8 | 908.3 | 1.00 |
cairo-native (embedded AOT) |
5230.1 ± 90.3 | 5083.0 | 5378.0 | 5.92 ± 0.14 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5282.3 ± 69.7 | 5177.2 | 5409.5 | 5.98 ± 0.13 |
Benchmark for program factorial_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.388 ± 0.051 | 5.292 | 5.428 | 1.00 |
cairo-native (embedded AOT) |
5.604 ± 0.090 | 5.455 | 5.731 | 1.04 ± 0.02 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.608 ± 0.060 | 5.486 | 5.698 | 1.04 ± 0.01 |
Benchmark for program fib_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.251 ± 0.029 | 5.189 | 5.288 | 1.04 ± 0.01 |
cairo-native (embedded AOT) |
5.043 ± 0.052 | 4.969 | 5.135 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.192 ± 0.086 | 5.099 | 5.320 | 1.03 ± 0.02 |
Benchmark for program linear_search
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
841.1 ± 5.0 | 831.7 | 848.5 | 1.00 |
cairo-native (embedded AOT) |
5159.4 ± 53.7 | 5082.3 | 5227.3 | 6.13 ± 0.07 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5205.2 ± 89.0 | 5092.9 | 5364.9 | 6.19 ± 0.11 |
Benchmark for program logistic_map
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
578.7 ± 6.3 | 568.0 | 587.7 | 1.00 |
cairo-native (embedded AOT) |
5251.1 ± 61.0 | 5150.2 | 5367.4 | 9.07 ± 0.14 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5537.5 ± 62.4 | 5413.2 | 5615.5 | 9.57 ± 0.15 |
This PR makes testing libfuncs quite complicated and its objective can be achieved by other, simpler, means. So, it has no meaning anymore.