cairo_native icon indicating copy to clipboard operation
cairo_native copied to clipboard
verified publisher icon lambdaclass

Implement libfunc profiler.

Open azteca1998 opened this issue 1 year ago • 3 comments

Implementation of a libfunc profiler.

For each executed libfunc, generates:

  • Number of samples.
  • Total number of ticks.
  • Average number of ticks.
  • Standard deviation.
  • Quartiles (0%, 25%, 50%, 75%, 100%).

Also implements an outlier detection system to filter out operating system noise.

Note: The x86_64 architecture handles context switches between cores correctly, but in aarch64 this is impossible. Results may not be reliable in aarch64.

Checklist

  • [ ] Linked to Github Issue
  • [ ] Unit tests added
  • [ ] Integration tests added.
  • [ ] This change requires new documentation.
    • [ ] Documentation has been added/updated.

azteca1998 avatar Nov 27 '24 23:11 azteca1998

Benchmarking results

Benchmark for program dict_insert

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 11.388 ± 0.061 11.329 11.498 2.27 ± 0.02
cairo-native (embedded AOT) 5.014 ± 0.021 4.980 5.050 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 5.113 ± 0.032 5.069 5.174 1.02 ± 0.01

Benchmark for program dict_snapshot

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 810.6 ± 8.0 798.6 822.4 1.00
cairo-native (embedded AOT) 4914.7 ± 27.8 4865.0 4957.1 6.06 ± 0.07
cairo-native (embedded JIT using LLVM's ORC Engine) 5027.0 ± 30.3 4987.5 5087.9 6.20 ± 0.07

Benchmark for program factorial_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.259 ± 0.019 5.229 5.290 1.00
cairo-native (embedded AOT) 5.305 ± 0.024 5.268 5.348 1.01 ± 0.01
cairo-native (embedded JIT using LLVM's ORC Engine) 5.374 ± 0.041 5.316 5.465 1.02 ± 0.01

Benchmark for program fib_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.116 ± 0.011 5.103 5.132 1.06 ± 0.01
cairo-native (embedded AOT) 4.835 ± 0.021 4.809 4.866 1.00
cairo-native (embedded JIT using LLVM's ORC Engine) 4.878 ± 0.024 4.835 4.910 1.01 ± 0.01

Benchmark for program linear_search

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 832.8 ± 4.6 824.3 838.6 1.00
cairo-native (embedded AOT) 4984.5 ± 19.6 4948.1 5011.7 5.99 ± 0.04
cairo-native (embedded JIT using LLVM's ORC Engine) 5110.3 ± 33.6 5069.3 5163.4 6.14 ± 0.05

Benchmark for program logistic_map

Open benchmarks
Command Mean [ms] Min [ms] Max [ms] Relative
Cairo-vm (Rust, Cairo 1) 539.1 ± 3.4 534.3 545.7 1.00
cairo-native (embedded AOT) 5025.6 ± 31.4 4965.9 5058.4 9.32 ± 0.08
cairo-native (embedded JIT using LLVM's ORC Engine) 5209.6 ± 24.5 5178.4 5257.8 9.66 ± 0.08

github-actions[bot] avatar Nov 27 '24 23:11 github-actions[bot]

Benchmark results Main vs HEAD.

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_insert.cairo (JIT) 5.177 ± 0.039 5.117 5.229 1.02 ± 0.02
base dict_insert.cairo (AOT) 5.084 ± 0.067 4.976 5.149 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_insert.cairo (JIT) 5.127 ± 0.019 5.092 5.151 1.01 ± 0.01
head dict_insert.cairo (AOT) 5.075 ± 0.066 5.018 5.230 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base dict_snapshot.cairo (JIT) 4.987 ± 0.029 4.922 5.018 1.02 ± 0.01
base dict_snapshot.cairo (AOT) 4.880 ± 0.033 4.824 4.920 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head dict_snapshot.cairo (JIT) 5.034 ± 0.025 5.006 5.082 1.02 ± 0.01
head dict_snapshot.cairo (AOT) 4.940 ± 0.023 4.909 4.992 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base factorial_2M.cairo (JIT) 5.447 ± 0.016 5.430 5.476 1.03 ± 0.00
base factorial_2M.cairo (AOT) 5.282 ± 0.018 5.252 5.306 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head factorial_2M.cairo (JIT) 5.403 ± 0.019 5.379 5.442 1.01 ± 0.01
head factorial_2M.cairo (AOT) 5.355 ± 0.034 5.306 5.403 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base fib_2M.cairo (JIT) 4.844 ± 0.024 4.801 4.879 1.01 ± 0.01
base fib_2M.cairo (AOT) 4.791 ± 0.042 4.741 4.876 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head fib_2M.cairo (JIT) 4.930 ± 0.030 4.885 4.976 1.01 ± 0.01
head fib_2M.cairo (AOT) 4.895 ± 0.029 4.838 4.928 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base linear_search.cairo (JIT) 5.092 ± 0.037 5.010 5.136 1.02 ± 0.01
base linear_search.cairo (AOT) 5.002 ± 0.053 4.913 5.073 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head linear_search.cairo (JIT) 5.136 ± 0.024 5.105 5.180 1.03 ± 0.01
head linear_search.cairo (AOT) 5.000 ± 0.022 4.956 5.035 1.00

Base

Command Mean [s] Min [s] Max [s] Relative
base logistic_map.cairo (JIT) 5.201 ± 0.076 5.109 5.318 1.05 ± 0.02
base logistic_map.cairo (AOT) 4.952 ± 0.038 4.916 5.029 1.00

Head

Command Mean [s] Min [s] Max [s] Relative
head logistic_map.cairo (JIT) 5.254 ± 0.029 5.202 5.288 1.04 ± 0.01
head logistic_map.cairo (AOT) 5.069 ± 0.052 4.988 5.137 1.00

github-actions[bot] avatar Nov 27 '24 23:11 github-actions[bot]

Codecov Report

Attention: Patch coverage is 85.04673% with 48 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@b92123c). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/libfuncs/starknet.rs 27.27% 32 Missing :warning:
src/libfuncs/bounded_int.rs 52.94% 8 Missing :warning:
src/libfuncs/cast.rs 72.72% 3 Missing :warning:
src/libfuncs/coupon.rs 0.00% 2 Missing :warning:
src/libfuncs/enum.rs 66.66% 2 Missing :warning:
src/libfuncs/bytes31.rs 75.00% 1 Missing :warning:
Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #946   +/-   ##
=======================================
  Coverage        ?   80.21%           
=======================================
  Files           ?      112           
  Lines           ?    29880           
  Branches        ?        0           
=======================================
  Hits            ?    23967           
  Misses          ?     5913           
  Partials        ?        0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Nov 27 '24 23:11 codecov-commenter

✅ Code is now correctly formatted.

github-actions[bot] avatar Jun 10 '25 21:06 github-actions[bot]

All comments were resolved

FrancoGiachetta avatar Jun 12 '25 20:06 FrancoGiachetta

These changes were squashed in #1253.

JulianGCalderon avatar Jun 17 '25 14:06 JulianGCalderon