Writing results of `Deno.bench` / checking against previous runs

[harrysolovay]

Is there a standard approach to utilizing Deno.bench to safeguard against performance degradation without manually comparing output with that of previous runs?

Currently, I execute deno bench bench.ts > bench_results and pipe its stdout into a file, which I then commit. Upon making changes, I re-run this command. I then manually compare changes to bench_results (in a given PR). Ideally, I would automate this process / turn it into a GH workflow / check. I could not find mention of a solution in the manual's Benchmarking section... and I'd rather not parse the bencher's output, as I'd imagine it may change in future versions of Deno. Any suggestions?

[bartlomieju]

Is there a standard approach to utilizing Deno.bench to safeguard against performance degradation without manually comparing output with that of previous runs?

Not yet, it was proposed by @evanwashere who wanted to provide a solution to this problem. But I'm not sure what's the status of that work.

[harrysolovay]

Related note for posterity: I see that the previous std approach to benching is deprecated. This approach allowed for in-memory access to the bench data (simplifies serializing that data and then checking against future runs). Anyways, I'm very excited to see how this API progresses!

[evanwashere]

There is a mock design for it, but it will likely only be implemented as a follow-up to #14305

[bartlomieju]

Does https://github.com/denoland/deno/pull/14305 solve this issue @harrysolovay?

[harrysolovay]

Somewhat. I was hoping for an API comparable to that of snapshot testing. I could imagine something along the lines of...

import { assertBenchGte } from "...";

Deno.test("Benchmarks are as good or better", (t) => {
  assertBenchGte(t, () => doTheThing());
});

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[harrysolovay]

Pinging this issue to stave off the stale bot.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[harrysolovay]

Warding off the stale bot