crypto-bigint icon indicating copy to clipboard operation
crypto-bigint copied to clipboard

Add gungraun constant-time benchmarks and checker

Open xyzzyz opened this issue 3 months ago • 2 comments

This is a proof of concept for using Gungraun for testing constant-timeness of the operations. The purpose of this PR is less so getting it merged, and more so gauging your interest in this work.

Gungraun uses Valgrind tooling to obtain accurate instruction counts for benchmark executions. This allows comparing runs on different inputs to see if the instruction counts are the same.

I added a number of Gungraun benchmarks, along with a simple binary to process the results (with the intention of using it in CI pipeline). Interestingly, I found that while Gungraun calculates the exact same instruction count for add/sub/mul and GCD operations, it finds tiny differences between mod_pow runs on different arguments (on the order of 0.005% difference in instruction counts).

Are you interested in including this in crypto-bigint test suite? If so, do you have any suggestions for next steps? Some of the ones I see are:

  • adding more benchmarks to get higher coverage
  • polishing it up and including in CI (for the purposes of constant timeness checking, but we could also get performance regression detection as side benefit)
  • investigate exactly why the instruction counts are different for mod_pow.

Thanks for your feedback!

xyzzyz avatar Nov 16 '25 23:11 xyzzyz

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 79.87%. Comparing base (9cbea3a) to head (c8c9bb3). :warning: Report is 136 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1005      +/-   ##
==========================================
- Coverage   79.87%   79.87%   -0.01%     
==========================================
  Files         163      163              
  Lines       17593    17593              
==========================================
- Hits        14053    14052       -1     
- Misses       3540     3541       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Nov 16 '25 23:11 codecov[bot]

@xyzzyz we're definitely interested in ways to better validate constant-time operation. It's something I've been very interested in working on, but also putting off until we can get to the next release.

I was also looking at cargo-checkct for this purpose: #796

tarcieri avatar Nov 23 '25 20:11 tarcieri