cairo_native icon indicating copy to clipboard operation
cairo_native copied to clipboard

Use SierraGenerator for testing (part 3)

Open FrancoGiachetta opened this issue 9 months ago • 3 comments

This PR changes some tests to use Sierra code, rather than Cairo code.

This has the following advantages:

  • Reduces test execution time by omitting compilation.
  • We ensure that the test is not modified by changes in the compiler

Related to #1152

Checklist

  • [x] Linked to Github Issue

FrancoGiachetta avatar Feb 28 '25 13:02 FrancoGiachetta

Benchmark results Main vs HEAD.

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_insert.cairo (JIT) 5.251 ± 0.018 5.208 5.275 1.01 ± 0.01
base dict_insert.cairo (AOT) 5.183 ± 0.041 5.126 5.273 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_insert.cairo (JIT) 5.285 ± 0.090 5.166 5.421 1.02 ± 0.02
head dict_insert.cairo (AOT) 5.185 ± 0.072 5.077 5.280 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_snapshot.cairo (JIT) 5.112 ± 0.033 5.052 5.154 1.02 ± 0.01
base dict_snapshot.cairo (AOT) 5.029 ± 0.027 4.983 5.069 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_snapshot.cairo (JIT) 5.122 ± 0.097 5.002 5.281 1.01 ± 0.03
head dict_snapshot.cairo (AOT) 5.057 ± 0.115 4.890 5.260 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base factorial_2M.cairo (JIT) 5.491 ± 0.035 5.445 5.543 1.01 ± 0.01
base factorial_2M.cairo (AOT) 5.424 ± 0.020 5.387 5.454 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head factorial_2M.cairo (JIT) 5.413 ± 0.106 5.285 5.604 1.00
head factorial_2M.cairo (AOT) 5.538 ± 0.141 5.345 5.730 1.02 ± 0.03

Base

Command Mean [s] Min [s] Max [s] Relative
base fib_2M.cairo (JIT) 5.034 ± 0.028 4.993 5.081 1.01 ± 0.01
base fib_2M.cairo (AOT) 4.986 ± 0.029 4.931 5.039 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head fib_2M.cairo (JIT) 5.036 ± 0.043 4.979 5.119 1.01 ± 0.02
head fib_2M.cairo (AOT) 5.000 ± 0.107 4.864 5.128 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base linear_search.cairo (JIT) 5.234 ± 0.042 5.203 5.333 1.02 ± 0.01
base linear_search.cairo (AOT) 5.142 ± 0.038 5.074 5.204 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head linear_search.cairo (JIT) 5.231 ± 0.099 5.092 5.428 1.03 ± 0.02
head linear_search.cairo (AOT) 5.096 ± 0.062 5.001 5.179 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base logistic_map.cairo (JIT) 5.338 ± 0.043 5.256 5.409 1.03 ± 0.01
base logistic_map.cairo (AOT) 5.196 ± 0.034 5.140 5.239 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head logistic_map.cairo (JIT) 5.344 ± 0.133 5.169 5.563 1.05 ± 0.03
head logistic_map.cairo (AOT) 5.101 ± 0.094 5.016 5.312 1.00

github-actions[bot] avatar Feb 28 '25 13:02 github-actions[bot]

Codecov Report

Attention: Patch coverage is 99.67267% with 2 lines in your changes missing coverage. Please review.

Project coverage is 80.77%. Comparing base (0a05bba) to head (c1f0d2d).

Files with missing lines Patch % Lines
src/values.rs 80.00% 2 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1127      +/-   ##
==========================================
+ Coverage   80.49%   80.77%   +0.27%     
==========================================
  Files         110      110              
  Lines       29605    29838     +233     
==========================================
+ Hits        23832    24101     +269     
+ Misses       5773     5737      -36     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Feb 28 '25 13:02 codecov-commenter

Benchmarking results

Benchmark for program dict_insert

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 11.508 ± 0.040 11.468 11.596 2.28 ± 0.02
cairo-native (embedded AOT) 5.042 ± 0.033 4.999 5.096 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 5.135 ± 0.063 5.078 5.304 1.02 ± 0.01

Benchmark for program dict_snapshot

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 816.6 ± 8.8 802.2 828.9 1.00
cairo-native (embedded AOT) 4875.0 ± 19.4 4854.2 4909.8 5.97 ± 0.07
cairo-native (embedded JIT using LLVM's ORC Engine) 4993.1 ± 20.5 4958.2 5023.5 6.11 ± 0.07

Benchmark for program factorial_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.310 ± 0.036 5.271 5.398 1.00
cairo-native (embedded AOT) 5.333 ± 0.018 5.298 5.360 1.00 ± 0.01
cairo-native (embedded JIT using LLVM's ORC Engine) 5.375 ± 0.043 5.321 5.459 1.01 ± 0.01

Benchmark for program fib_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.106 ± 0.033 5.078 5.191 1.05 ± 0.01
cairo-native (embedded AOT) 4.848 ± 0.036 4.802 4.916 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 4.874 ± 0.027 4.836 4.915 1.01 ± 0.01

Benchmark for program linear_search

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 835.2 ± 5.9 829.7 844.4 1.00
cairo-native (embedded AOT) 4976.1 ± 83.6 4895.7 5196.2 5.96 ± 0.11
cairo-native (embedded JIT using LLVM's ORC Engine) 5095.0 ± 44.8 5062.7 5204.6 6.10 ± 0.07

Benchmark for program logistic_map

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 542.2 ± 4.3 533.6 549.9 1.00
cairo-native (embedded AOT) 5005.5 ± 41.8 4951.7 5087.8 9.23 ± 0.11
cairo-native (embedded JIT using LLVM's ORC Engine) 5198.1 ± 55.2 5167.2 5352.8 9.59 ± 0.13

github-actions[bot] avatar Feb 28 '25 13:02 github-actions[bot]

This PR makes testing libfuncs quite complicated and its objective can be achieved by other, simpler, means. So, it has no meaning anymore.

FrancoGiachetta avatar Sep 16 '25 16:09 FrancoGiachetta