Better Benchmarking for openapi-fetch?

[gzm0]

@drwpow, I'm taking the liberty to moving your comment to an issue so we do not lose the discussion.

Also a note—I’ve found the benchmarking to be wildly inconsistent 😓. It’s still in beta for Vitest, and I think we should have another alternative.

Or, put another way, the absolute numbers fluctuate wildly. But the relative measurements have some consistency (i.e. openapi-fetch being XX% faster than another package stays consistent-ish).

It’s been on my list to come up with more reliable benchmarks—would love your thoughts if there’s a better way to profile performance.

Also like I said before, anecdotally I haven’t had good experiences with performance regression tests (or, at least, not without an entire team at a company dedicated to maintaining them, and we don’t have that for this OSS project). CI (especially free CI) just fluctuates too widely with resources to be reliable. But I’m strongly in favor of manually-run benchmarking, that can be run when we want to see if something moves the needle up or down. And I think there’s improvements to be made in this area here, and I welcome your input! 🙏

Originally posted by @drwpow in https://github.com/openapi-ts/openapi-typescript/issues/1810#issuecomment-2265690346