model-transparency icon indicating copy to clipboard operation
model-transparency copied to clipboard

Published 20 hours ago • verified publisher icon sigstore

Benchmarking scripts

Open laurentsimon opened this issue 1 year ago • 3 comments

We'd like to have benchmarking scripts to improve on the current shell https://github.com/sigstore/model-transparency/blob/main/model_signing/benchmarks/run.sh

We welcome contributions; this is a really important piece to be able to compare various signing options (serialization, hash + params) and be able to cut a v1 release.

@mihaimaruseac has written some python-based script we can work off.

@font FYI

laurentsimon avatar Jun 12 '24 16:06 laurentsimon

@mihaimaruseac has written some python-based script we can work off.

@mihaimaruseac Are you able to share these?

font avatar Jun 12 '24 18:06 font

I'm debating between sharing the scripts as I used them for previous benchmarks or adapting them to the new API (under work) and having them then act as CI checks

mihaimaruseac avatar Jun 13 '24 00:06 mihaimaruseac

+1 on adapting them to the new APIs. Sharing the script may speed up adapting them to the new API, if someone's interested in doing the work

laurentsimon avatar Jun 13 '24 17:06 laurentsimon

We should make sure to benchmark Python 3.10 vs Python 3.11 / 3.12. I'm curious if the hashlib.file_digest polyfill (#300) will show a meaningful difference, as there were 1-2 optimizations in the upstream that weren't included.

spencerschrock avatar Aug 29 '24 16:08 spencerschrock