cairo_native icon indicating copy to clipboard operation
cairo_native copied to clipboard

Use SierraGenerator for testing (part 5)

Open FrancoGiachetta opened this issue 9 months ago • 3 comments

This PR changes some tests to use Sierra code, rather than Cairo code.

This has the following advantages:

  • Reduces test execution time by omitting compilation.
  • We ensure that the test is not modified by changes in the compiler

Related to #1152

Checklist

  • [x] Linked to Github Issue

FrancoGiachetta avatar Feb 28 '25 13:02 FrancoGiachetta

Benchmark results Main vs HEAD.

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_insert.cairo (JIT) 5.128 ± 0.068 5.048 5.287 1.02 ± 0.01
base dict_insert.cairo (AOT) 5.036 ± 0.018 5.005 5.069 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_insert.cairo (JIT) 5.100 ± 0.037 5.062 5.191 1.02 ± 0.01
head dict_insert.cairo (AOT) 4.980 ± 0.028 4.936 5.028 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_snapshot.cairo (JIT) 5.012 ± 0.029 4.975 5.067 1.02 ± 0.01
base dict_snapshot.cairo (AOT) 4.917 ± 0.034 4.870 4.972 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_snapshot.cairo (JIT) 5.008 ± 0.041 4.957 5.092 1.02 ± 0.02
head dict_snapshot.cairo (AOT) 4.919 ± 0.074 4.842 5.062 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base factorial_2M.cairo (JIT) 5.378 ± 0.031 5.318 5.412 1.01 ± 0.01
base factorial_2M.cairo (AOT) 5.330 ± 0.034 5.293 5.392 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head factorial_2M.cairo (JIT) 5.379 ± 0.025 5.340 5.417 1.01 ± 0.01
head factorial_2M.cairo (AOT) 5.313 ± 0.045 5.253 5.381 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base fib_2M.cairo (JIT) 4.924 ± 0.049 4.822 4.983 1.02 ± 0.01
base fib_2M.cairo (AOT) 4.841 ± 0.036 4.805 4.917 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head fib_2M.cairo (JIT) 4.883 ± 0.030 4.824 4.928 1.01 ± 0.01
head fib_2M.cairo (AOT) 4.831 ± 0.036 4.762 4.888 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base linear_search.cairo (JIT) 5.163 ± 0.048 5.084 5.236 1.04 ± 0.02
base linear_search.cairo (AOT) 4.984 ± 0.070 4.890 5.132 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head linear_search.cairo (JIT) 5.044 ± 0.012 5.022 5.062 1.03 ± 0.01
head linear_search.cairo (AOT) 4.910 ± 0.021 4.876 4.936 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base logistic_map.cairo (JIT) 5.206 ± 0.042 5.154 5.280 1.03 ± 0.01
base logistic_map.cairo (AOT) 5.044 ± 0.027 5.013 5.109 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head logistic_map.cairo (JIT) 5.147 ± 0.030 5.107 5.207 1.03 ± 0.01
head logistic_map.cairo (AOT) 4.985 ± 0.018 4.961 5.013 1.00

github-actions[bot] avatar Feb 28 '25 14:02 github-actions[bot]

Codecov Report

Attention: Patch coverage is 95.00000% with 15 lines in your changes missing coverage. Please review.

Project coverage is 80.64%. Comparing base (0a05bba) to head (8ee859c).

Files with missing lines Patch % Lines
src/values.rs 88.78% 12 Missing :warning:
src/libfuncs/felt252_dict_entry.rs 96.66% 2 Missing :warning:
src/libfuncs/felt252.rs 97.36% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1129      +/-   ##
==========================================
+ Coverage   80.49%   80.64%   +0.14%     
==========================================
  Files         110      110              
  Lines       29605    29776     +171     
==========================================
+ Hits        23832    24014     +182     
+ Misses       5773     5762      -11     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Feb 28 '25 14:02 codecov-commenter

Benchmarking results

Benchmark for program dict_insert

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 11.792 ± 0.080 11.655 11.892 2.33 ± 0.04
cairo-native (embedded AOT) 5.070 ± 0.080 4.977 5.246 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 5.144 ± 0.045 5.077 5.223 1.01 ± 0.02

Benchmark for program dict_snapshot

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 883.1 ± 15.2 859.8 908.3 1.00
cairo-native (embedded AOT) 5230.1 ± 90.3 5083.0 5378.0 5.92 ± 0.14
cairo-native (embedded JIT using LLVM's ORC Engine) 5282.3 ± 69.7 5177.2 5409.5 5.98 ± 0.13

Benchmark for program factorial_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.388 ± 0.051 5.292 5.428 1.00
cairo-native (embedded AOT) 5.604 ± 0.090 5.455 5.731 1.04 ± 0.02
cairo-native (embedded JIT using LLVM's ORC Engine) 5.608 ± 0.060 5.486 5.698 1.04 ± 0.01

Benchmark for program fib_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.251 ± 0.029 5.189 5.288 1.04 ± 0.01
cairo-native (embedded AOT) 5.043 ± 0.052 4.969 5.135 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 5.192 ± 0.086 5.099 5.320 1.03 ± 0.02

Benchmark for program linear_search

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 841.1 ± 5.0 831.7 848.5 1.00
cairo-native (embedded AOT) 5159.4 ± 53.7 5082.3 5227.3 6.13 ± 0.07
cairo-native (embedded JIT using LLVM's ORC Engine) 5205.2 ± 89.0 5092.9 5364.9 6.19 ± 0.11

Benchmark for program logistic_map

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 578.7 ± 6.3 568.0 587.7 1.00
cairo-native (embedded AOT) 5251.1 ± 61.0 5150.2 5367.4 9.07 ± 0.14
cairo-native (embedded JIT using LLVM's ORC Engine) 5537.5 ± 62.4 5413.2 5615.5 9.57 ± 0.15

github-actions[bot] avatar Feb 28 '25 14:02 github-actions[bot]

This PR makes testing libfuncs quite complicated and its objective can be achieved by other, simpler, means. So, it has no meaning anymore.

FrancoGiachetta avatar Sep 16 '25 16:09 FrancoGiachetta