nng
nng copied to clipboard
nanomsg
CI benchmarks
v1.4 was mostly about perf optimisations (awesome to see you focusing on that, btw), but there isn't any concrete indication of what improvements we can expect. If throughput/latency/whatever is improved it would be interesting to have a simple, exemplary before/after perf test. Would also help catch perf regressions.
This post came to mind, but I honestly haven't researched what other/better options exist.
Also, there's perf issues raised on occasion and people could supply a PR adding to the test suite.
I'm not sure that GitHub actions is really the best place to test performance. You really want to have a control, and I don't know if the environment is stable enough from one run to the next to use for this without causing a lot of bogus complaints about regressions.
My experience so far is that GH is not acceptable for benchmarking. You really need bare metal or at least dedicate CPUs and memory to isolate from noisy neighbors. It might be possible to pay for that, but this project doesn't have a budget for it.
I'm closing this for now, because I don't think we can meaningfully do it on GH. And I have no better location for CI/CD.