Future-proof Benchmarking ⏱

[p-shahi]

eta: 2023Q2

• [ ] perf protocol
  • [x] spec https://github.com/libp2p/specs/pull/478
  • [x] https://github.com/libp2p/rust-libp2p/pull/3508 (rust-libp2p)
  • [x] go-libp2p https://github.com/libp2p/test-plans/pull/184
  • [x] https://github.com/libp2p/js-libp2p/pull/1604 (js-libp2p)
  • [x] https://github.com/libp2p/test-plans/issues/261
  • [ ] #204
• [ ] Create performance tooling to run perf protocol implementation on cloud instances
  • [x] First iteration https://github.com/libp2p/test-plans/pull/184
  • [x] Only use a single seconds measurement. Don't split in connection establishment, upload and download. These are difficult to differentiate. https://github.com/libp2p/test-plans/pull/207
  • [x] Add js-libp2p https://github.com/libp2p/js-libp2p/pull/1604 https://github.com/libp2p/test-plans/pull/244
  • [ ] Add back zig-libp2p
  • [ ] Run iperf on both UDP and TCP
  • [ ] Requests per second (for some request/response protocol)
  • [ ] Handshakes per second (useful to identify overhead in connection initialization).
  • [ ] Support testing other security protocols, i.e. Noise
  • [ ] Resource utilization benchmark?
  • [ ] Don't blindly trust server and clients ssh key (mitm attack)
• [x] Automation see https://github.com/libp2p/test-plans/issues/183
• [x] Visualize data emitted by above performance tooling #156
  • [x] ideally mirror advanced filtering in the URL for better sharing
• [ ] https://github.com/libp2p/test-plans/issues/27
• [x] https://github.com/libp2p/test-plans/issues/65
• [ ] https://github.com/libp2p/blog/issues/74
• [ ] Diverse benchmarks e.g. multiple streams or multiple connections
• [ ] Write documentation

[BigLep]

@mxinden : I know you're doing blog drafting at https://observablehq.com/@mxinden-workspace/libp2p-perf - thanks!

I did leave some comments inline.

A couple of general things that came to mind while looking at it:

1. Is this just where we're drafting, or something we expect to point users to?
2. If pointing users to it, can we create a libp2p Observable namespace?
3. What is going to be the presence on the libp2p blog about this?

(no pressure to respond now - I don't mean to be catching you in-progress. If this has been though through and you just haven't gotten to any of this yet, that is fine. I am just leaving notes now while it's coming to mind.)

[mxinden]

Status Update

https://github.com/libp2p/test-plans/pull/184 and thus a first iteration of the libp2p performance benchmark setup is merged. Follow-up / future work in tracked in the description of this issue.

[mxinden]

Crossposting a Slack message here, for those following through GitHub only.

First iteration of the libp2p performance benchmarking setup merged

More specifically https://github.com/libp2p/test-plans/pull/184.

Let's try it out. We recently released a new version of rust-libp2p, v0.52.0. Now we can add it to the benchmarking setup, see how well it performs.

1. Create a pull request against libp2p/test-plans adding the new version https://github.com/libp2p/test-plans/pull/202. (Note that in this particular case we are updating an existing version, as we have been already testing a pre-release of v0.52.0.)
2. Trigger the libp2p perf test GitHub workflow for the branch of the pull request https://github.com/libp2p/test-plans/actions/workflows/perf.yml
3. Follow the workflow run, which eventually pushes the benchmark results to the branch https://github.com/libp2p/test-plans/actions/runs/5351995316/jobs/9706612810
4. See the new commit on your pull request https://github.com/libp2p/test-plans/pull/202 https://github.com/libp2p/test-plans/pull/202/commits/b94960f74d0a2d85b429c311295721bc6d3f6e9f
5. Visualize the benchmark results via our ObservableHQ dashboard. Just make sure to choose the right branch, in this case perf-rust-libp2p-v0.52 in the Branch input https://observablehq.com/@mxinden-workspace/libp2p-performance-dashboard

Note that this is the first iteration only. There are lots of follow-ups, especially around the dashboard. Future work tracked on https://github.com/libp2p/test-plans/issues/63.

Thanks to Marco, Piotr and Marten for all the help on the project!

https://filecoinproject.slack.com/archives/C03K82MU486/p1687492998525109

[mxinden]

Status Update

ideally mirror advanced filtering in the URL for better sharing

You can now embed the repository and branch of the benchmark data in the dashboard URL. For example the dashboard link below displays the benchmarking data of https://github.com/libp2p/test-plans/pull/241.

https://observablehq.com/@libp2p-workspace/performance-dashboard?branch=perf-iperf-tcp