express icon indicating copy to clipboard operation
express copied to clipboard

Benchmarking suite documentation and historical benchmark storage

Open notthelewis opened this issue 1 year ago • 0 comments

This is the first time I've ever commited to a public repository, so I hope I've ticked all the right boxes!

I wanted to get stuck in and try to see if I could optimise anything, for curiosity's sake. I was delighted when I found the benchmarks folder but slightly disheartened when I couldn't find any documentation.

This PR attempts to address the documentation issue but also adds new functionality: Historical benchmark storage.

The idea is to store every benchmark output in plain text, sorted by version number and stored alongside the CPU and username of the person who ran the benchmark. The latter two points I'm a little unsure of, as I don't know whether it could be a security issue to store the username and CPU model of everyone that runs the benchmark.

Alongside this, I'm a little unsure of the benefit of storing all benchmarks in the repository itself. Might it be a better idea to add them to a gitignore, and use them only for local reference- especially since results are going to be extremely hardware dependent?

I would like to take this barebones feature and mould it into a more cohesive reporting suite, which allows you to compare the current output with previous outputs- or compare any two outputs by taking in the two files.

Before venturing too far down that path, I thought it best to try get a P.O.C in first and see whether this actually provides any value. Looking forward to hearing back :)

notthelewis avatar Apr 22 '23 22:04 notthelewis