mbrezu
Feat request: extended benchmarks and tests
Hello, I would like to request more benchmarks and tests to be added. I want to use this library, but I need to make sure it's stable in edgecases. For something low level like this, especially when using third party deps, testing is crucial for rapport and maintenance. I'm fine submitting a PR aswell, if you're open to contributions. I could add one test at a time, or everything at once. Or I can do test first, then benchmarks. However is most convenient for you.
Side Note: I do also want to make sure this library is compatible with clasp, as I think that would be potentially interesting, but this is more of a personal interest of mine, but maybe worth noting.
Let me know. Thank you,
Well, yes, extending the test script and adding benchmarks via a PR is a good idea.
After coming back and reviewing I think the tests are actually pretty good, but the benchmarks could be expanded for sure.
Also: I did mention getting this library portable, but, honestly I think the only way to do that is to use rust or c shims for the implementation specific primitives which I think would be impractical.
In anycase, if someone wants to use messagepack for performance, they're probably using SBCL anyways. And with clasp you could probably just extend with the c++ messagepack lib, but I digress.
--
My contribution will lag behind a little bit. I expected to start the week I submitted this issue, but it might take me another month before I have the time to allocate here.
@phmarek What are your thoughts on adding CSV logging and/or graphing for benchmarks in cl-messagepack? The idea would be to log the benchmark results as CSV. Then, for visualization, we could optionally generate graphs using either R (via ggplot2) or gnuplot. I’ve done something similar in cl-freelock if you’d like to see an example. If we go the R route, the script could be excluded via .gitattributes to avoid adding language noise to the GitHub page. The maintenance cost should be effectively zero. On the other hand, gnuplot might have the potential to be a bit more Lisp-native since generating gnuplot files is essentially just generating text. Do you think something like this would be a good fit for the project? If not, that’s fine with me, since it’s not technically necessary. It’s more of an aesthetic and convenience thing.
I'm fine with R, I use it myself for visualizations. (Though mostly with theme_tufte, here's why.)
The benchmark script is fine on the main branch; the only point I'm pondering is whether to put the R (and your CSV measurements, and a PNG visualization?) on eg. a benchmark branch (that could be merged or rebased when new measurements are taken).