js-libp2p icon indicating copy to clipboard operation
js-libp2p copied to clipboard

perf: Integrate unit and spec benchmarking in js-libp2p CI

Open SgtPooki opened this issue 2 years ago • 4 comments

discussed in https://pl-strflt.notion.site/js-libp2p-perf-benchmarking-f7c42f96cbd44799a321347f6c15c8d4?pvs=4

Goals

  • We would like to be able to benchmark js-libp2p across releases and compare performance, particularly across different permutations in the stack.
    • Testing different transports against each other performance wise.
  • We are currently having an issue with performance with [Yamux performance](https://github.com/ChainSafe/js-libp2p-yamux/issues/42) and we would like to be able to nail down why there are such drastic differences.
  • We have issues [sending large files](https://github.com/libp2p/js-libp2p/actions/runs/6265587906/job/17014869561?pr=2048#step:5:2415) and whilst this is cause by separate issue it would be beneficial to measure file transfer performance and have that running on CI.
### Tasks
- [ ] https://github.com/libp2p/js-libp2p/issues/1964
- [ ] https://github.com/libp2p/js-libp2p/issues/2120
- [ ] https://github.com/libp2p/test-plans/issues/307

SgtPooki avatar Oct 03 '23 18:10 SgtPooki

@maschad there was an action item for you in the Notion doc to create an issue for something (that wasn't captured well). pinging you here to request cleanup/refining of this issue (or close and point to issue you may have created?)

SgtPooki avatar Oct 03 '23 18:10 SgtPooki

Thanks for creating this @SgtPooki This issue was created https://github.com/libp2p/test-plans/issues/307 in test plans, I've also updated the description.

maschad avatar Oct 03 '23 23:10 maschad

Would need a place to store the benchmark data. Would be nice to have this on every CI pass so that we could see trends in performance. Requires a self-hosted runner since it would be part of the CI pipe and this is a performance metrics test.

dhuseby avatar May 07 '24 15:05 dhuseby

@achingbrain suggested adding this to the existing self-hosting runner currently doing transport interrupt tests.

dhuseby avatar May 07 '24 15:05 dhuseby