cairo_native icon indicating copy to clipboard operation
cairo_native copied to clipboard

Use SierraGenerator for testing (part 2)

Open FrancoGiachetta opened this issue 9 months ago • 3 comments

This PR changes some tests to use Sierra code, rather than Cairo code.

This has the following advantages:

  • Reduces test execution time by omitting compilation.
  • We ensure that the test is not modified by changes in the compiler

Related to #1152

Checklist

  • [x] Linked to Github Issue

FrancoGiachetta avatar Feb 28 '25 12:02 FrancoGiachetta

Benchmark results Main vs HEAD.

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_insert.cairo (JIT) 5.066 ± 0.040 5.014 5.139 1.01 ± 0.01
base dict_insert.cairo (AOT) 5.026 ± 0.043 4.970 5.115 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_insert.cairo (JIT) 5.246 ± 0.024 5.208 5.275 1.01 ± 0.01
head dict_insert.cairo (AOT) 5.183 ± 0.027 5.141 5.242 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_snapshot.cairo (JIT) 5.071 ± 0.027 5.033 5.120 1.02 ± 0.01
base dict_snapshot.cairo (AOT) 4.948 ± 0.059 4.844 5.028 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_snapshot.cairo (JIT) 5.143 ± 0.046 5.072 5.220 1.03 ± 0.02
head dict_snapshot.cairo (AOT) 4.981 ± 0.059 4.883 5.078 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base factorial_2M.cairo (JIT) 5.458 ± 0.057 5.322 5.520 1.03 ± 0.01
base factorial_2M.cairo (AOT) 5.311 ± 0.021 5.276 5.337 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head factorial_2M.cairo (JIT) 5.463 ± 0.060 5.376 5.548 1.01 ± 0.01
head factorial_2M.cairo (AOT) 5.407 ± 0.047 5.321 5.487 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base fib_2M.cairo (JIT) 4.929 ± 0.039 4.878 4.988 1.00 ± 0.01
base fib_2M.cairo (AOT) 4.914 ± 0.053 4.843 5.006 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head fib_2M.cairo (JIT) 4.936 ± 0.046 4.870 5.007 1.00 ± 0.01
head fib_2M.cairo (AOT) 4.935 ± 0.055 4.862 5.014 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base linear_search.cairo (JIT) 5.178 ± 0.033 5.136 5.216 1.04 ± 0.01
base linear_search.cairo (AOT) 4.993 ± 0.036 4.941 5.040 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head linear_search.cairo (JIT) 5.124 ± 0.082 5.005 5.228 1.04 ± 0.02
head linear_search.cairo (AOT) 4.929 ± 0.047 4.861 5.019 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base logistic_map.cairo (JIT) 5.296 ± 0.074 5.202 5.424 1.04 ± 0.02
base logistic_map.cairo (AOT) 5.098 ± 0.042 5.049 5.181 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head logistic_map.cairo (JIT) 5.178 ± 0.040 5.124 5.254 1.04 ± 0.01
head logistic_map.cairo (AOT) 4.966 ± 0.026 4.931 5.004 1.00

github-actions[bot] avatar Feb 28 '25 13:02 github-actions[bot]

Codecov Report

Attention: Patch coverage is 99.10448% with 3 lines in your changes missing coverage. Please review.

Project coverage is 80.72%. Comparing base (0a05bba) to head (b746727).

Files with missing lines Patch % Lines
src/values.rs 70.00% 3 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1126      +/-   ##
==========================================
+ Coverage   80.49%   80.72%   +0.22%     
==========================================
  Files         110      110              
  Lines       29605    29799     +194     
==========================================
+ Hits        23832    24054     +222     
+ Misses       5773     5745      -28     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Feb 28 '25 13:02 codecov-commenter

Benchmarking results

Benchmark for program dict_insert

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 11.445 ± 0.054 11.384 11.571 2.33 ± 0.02
cairo-native (embedded AOT) 4.914 ± 0.037 4.875 4.992 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 5.008 ± 0.034 4.951 5.050 1.02 ± 0.01

Benchmark for program dict_snapshot

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 795.6 ± 5.8 787.3 806.1 1.00
cairo-native (embedded AOT) 4796.1 ± 33.1 4754.9 4844.5 6.03 ± 0.06
cairo-native (embedded JIT using LLVM's ORC Engine) 4936.8 ± 31.1 4885.5 5005.6 6.21 ± 0.06

Benchmark for program factorial_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.301 ± 0.013 5.283 5.318 1.01 ± 0.01
cairo-native (embedded AOT) 5.231 ± 0.026 5.190 5.260 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 5.284 ± 0.023 5.242 5.320 1.01 ± 0.01

Benchmark for program fib_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.104 ± 0.017 5.080 5.133 1.07 ± 0.01
cairo-native (embedded AOT) 4.775 ± 0.028 4.730 4.824 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 4.810 ± 0.027 4.771 4.855 1.01 ± 0.01

Benchmark for program linear_search

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 819.6 ± 5.8 813.4 830.3 1.00
cairo-native (embedded AOT) 4869.4 ± 30.5 4830.8 4909.8 5.94 ± 0.06
cairo-native (embedded JIT using LLVM's ORC Engine) 5001.6 ± 20.7 4969.2 5039.7 6.10 ± 0.05

Benchmark for program logistic_map

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 535.5 ± 3.8 530.5 542.4 1.00
cairo-native (embedded AOT) 4933.3 ± 29.6 4872.7 4972.5 9.21 ± 0.09
cairo-native (embedded JIT using LLVM's ORC Engine) 5078.2 ± 23.7 5034.6 5107.4 9.48 ± 0.08

github-actions[bot] avatar Feb 28 '25 13:02 github-actions[bot]

This PR makes testing libfuncs quite complicated and its objective can be achieved by other, simpler, means. So, it has no meaning anymore.

FrancoGiachetta avatar Sep 16 '25 16:09 FrancoGiachetta