vitest
vitest copied to clipboard
vitest-dev
Vitest bench bug: NaNx faster than [alternative]
Describe the bug
Problem
Shows NaNx faster than ... which is wrong
Result
RERUN matchit/native/index.js x11
Filename pattern: .
✓ static.bench.js > matching [zero] 40967ms
name hz min max mean p75 p99 p995 p999 rme samples
· fast-node-parse 30,019,509.04 0.0000 0.0458 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.08% 15009755
· fast-path-parse[-runtime] 30,418,133.33 0.0000 0.0507 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.10% 15209067
· fast-path-parse[-aot] 30,350,035.33 0.0000 0.0383 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.08% 15175018
· hono-regex-match 30,830,529.52 0.0000 1.0265 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.42% 15415266
· hono-trie-match 5,601,067.35 0.0000 0.7059 0.0002 0.0002 0.0005 0.0005 0.0015 ±1.61% 2800534
· find-my-way 21,003,598.28 0.0000 0.6267 0.0000 0.0000 0.0003 0.0003 0.0005 ±1.86% 10501800
· wired 29,853,646.52 0.0000 1.2337 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.62% 14926824
· extreme-router 30,935,569.82 0.0000 0.2301 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.15% 15467786
· matchit-native 7,451,079.27 0.0000 0.5605 0.0001 0.0001 0.0004 0.0004 0.0006 ±0.39% 3725540
· matchit-wasm 20,943,997.97 0.0000 0.0403 0.0000 0.0000 0.0001 0.0001 0.0001 ±0.06% 10471999
✓ static.bench.js > parse [zero] 34738ms
name hz min max mean p75 p99 p995 p999 rme samples
· fast-node-parse 31,905,545.74 0.0000 0.2185 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.14% 15952773
· fast-path-parse[-runtime] 30,474,755.27 0.0000 0.4835 0.0000 0.0000 0.0000 0.0001 0.0003 ±0.34% 15237378
· fast-path-parse[-aot] 30,550,115.28 0.0000 0.5365 0.0000 0.0000 0.0000 0.0001 0.0003 ±0.43% 15275058
· hono-regex-match 30,748,886.77 0.0000 0.0305 0.0000 0.0000 0.0000 0.0000 0.0000 ±0.06% 15374444
· hono-trie-match 5,450,266.45 0.0000 0.5379 0.0002 0.0002 0.0004 0.0005 0.0009 ±1.48% 2725134
· find-my-way 19,740,240.53 0.0000 0.5548 0.0001 0.0000 0.0002 0.0003 0.0005 ±1.67% 9870125
· extreme-router 31,241,234.06 0.0000 0.0358 0.0000 0.0000 0.0000 0.0000 0.0000 ±0.07% 15620618
· matchit-native 6,151,373.16 0.0000 0.5451 0.0002 0.0002 0.0005 0.0005 0.0008 ±1.13% 3075687
· matchit-wasm 9,976,179.86 0.0000 0.5622 0.0001 0.0001 0.0002 0.0003 0.0005 ±0.79% 4988090
✓ static.bench.js > matching [zero] 38387ms
name hz min max mean p75 p99 p995 p999 rme samples
· fast-node-parse 28,124,198.59 0.0000 1.8666 0.0000 0.0000 0.0000 0.0001 0.0003 ±0.99% 14062100
· fast-path-parse[-runtime] 31,349,635.64 0.0000 0.0574 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.10% 15674818
· fast-path-parse[-aot] 30,679,080.65 0.0000 0.4049 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.24% 15339541
· hono-regex-match 31,473,079.80 0.0000 0.0644 0.0000 0.0000 0.0000 0.0000 0.0001 ±0.12% 15736541
· hono-trie-match 5,039,334.34 0.0000 0.5752 0.0002 0.0002 0.0005 0.0005 0.0015 ±1.66% 2519668
· find-my-way 16,754,992.33 0.0000 0.5524 0.0001 0.0000 0.0003 0.0003 0.0005 ±1.48% 8382641
· wired 15,222,495.72 0.0000 0.5142 0.0001 0.0001 0.0001 0.0001 0.0037 ±0.28% 7611249
· extreme-router 23,429,667.77 0.0000 0.5260 0.0000 0.0000 0.0001 0.0002 0.0003 ±0.38% 11714834
· matchit-native 7,768,851.08 0.0000 0.5821 0.0001 0.0001 0.0005 0.0006 0.0009 ±0.99% 3884426
· matchit-wasm 18,727,100.69 0.0000 0.0460 0.0001 0.0001 0.0001 0.0001 0.0001 ±0.08% 9363551
✓ static.bench.js > matching [easy] 31077ms
name hz min max mean p75 p99 p995 p999 rme samples
· fast-node-parse 27,382,696.63 0.0000 0.5385 0.0000 0.0000 0.0000 0.0001 0.0003 ±0.45% 13691349
· fast-path-parse[-runtime] 20,995,865.16 0.0000 3.7429 0.0000 0.0000 0.0001 0.0001 0.0003 ±1.96% 10497933
· fast-path-parse[-aot] 21,158,889.41 0.0000 0.6327 0.0000 0.0000 0.0001 0.0001 0.0003 ±1.67% 10579445
· hono-regex-match 11,774,569.36 0.0000 0.7902 0.0001 0.0001 0.0002 0.0003 0.0005 ±1.53% 5891585
· hono-trie-match 2,302,971.61 0.0002 0.6820 0.0004 0.0004 0.0009 0.0010 0.0031 ±1.53% 1151486
· find-my-way 12,918,306.94 0.0000 0.6637 0.0001 0.0001 0.0003 0.0003 0.0005 ±1.55% 6459154
· wired 18,126,992.84 0.0000 0.0683 0.0001 0.0000 0.0001 0.0001 0.0036 ±0.23% 9063497
· extreme-router 25,975,278.12 0.0000 0.6053 0.0000 0.0000 0.0000 0.0001 0.0003 ±1.37% 12987640
· matchit-native 7,970,656.72 0.0000 0.6431 0.0001 0.0001 0.0005 0.0006 0.0009 ±0.79% 3988258
· matchit-wasm 17,297,253.31 0.0000 0.0595 0.0001 0.0001 0.0001 0.0001 0.0002 ±0.13% 8648627
✓ static.bench.js > matching [easy] 33384ms
name hz min max mean p75 p99 p995 p999 rme samples
· fast-node-parse 27,961,393.90 0.0000 0.5879 0.0000 0.0000 0.0000 0.0001 0.0003 ±0.63% 13980697
· fast-path-parse[-runtime] 20,679,474.78 0.0000 0.6838 0.0000 0.0000 0.0001 0.0001 0.0003 ±1.24% 10339738
· fast-path-parse[-aot] 23,113,812.51 0.0000 0.4921 0.0000 0.0000 0.0001 0.0001 0.0003 ±0.61% 11556907
· hono-regex-match 13,817,366.20 0.0000 0.6592 0.0001 0.0001 0.0001 0.0002 0.0004 ±1.68% 6908684
· hono-trie-match 3,532,856.34 0.0001 0.8187 0.0003 0.0003 0.0005 0.0006 0.0019 ±1.76% 1766429
· find-my-way 13,694,773.15 0.0000 0.6684 0.0001 0.0001 0.0003 0.0003 0.0005 ±1.92% 6847387
· wired 30,541,689.90 0.0000 0.4797 0.0000 0.0000 0.0000 0.0000 0.0002 ±0.21% 15270845
· extreme-router 21,367,267.06 0.0000 0.6366 0.0000 0.0000 0.0001 0.0002 0.0003 ±1.22% 10683634
· matchit-native 8,110,694.23 0.0000 0.5349 0.0001 0.0001 0.0005 0.0006 0.0008 ±0.82% 4056958
· matchit-wasm 17,752,789.22 0.0000 0.0355 0.0001 0.0001 0.0001 0.0001 0.0002 ±0.08% 8876395
✓ static.bench.js > matching [medium] 25891ms
name hz min max mean p75 p99 p995 p999 rme samples
· fast-node-parse 27,174,588.37 0.0000 0.5482 0.0000 0.0000 0.0000 0.0001 0.0003 ±0.29% 13587295
· fast-path-parse[-runtime] 14,788,582.43 0.0000 0.7382 0.0001 0.0001 0.0001 0.0002 0.0003 ±0.66% 7394292
· fast-path-parse[-aot] 17,220,880.60 0.0000 0.6600 0.0001 0.0001 0.0001 0.0002 0.0004 ±1.30% 8610441
· hono-regex-match 12,172,305.03 0.0000 0.6217 0.0001 0.0001 0.0002 0.0003 0.0005 ±0.91% 6086153
· hono-trie-match 1,362,665.99 0.0005 0.7682 0.0007 0.0007 0.0014 0.0035 0.0091 ±1.18% 1000000
· find-my-way 5,730,973.65 0.0000 5.2088 0.0002 0.0002 0.0005 0.0006 0.0018 ±2.89% 2865487
· wired 11,709,312.53 0.0000 0.6307 0.0001 0.0001 0.0001 0.0002 0.0039 ±0.33% 5854657
· extreme-router 21,637,816.40 0.0000 0.6980 0.0000 0.0000 0.0001 0.0002 0.0004 ±1.38% 10818909
· matchit-native 6,738,257.77 0.0000 0.7877 0.0001 0.0001 0.0006 0.0008 0.0011 ±1.40% 3369129
· matchit-wasm 11,742,427.86 0.0000 0.1590 0.0001 0.0001 0.0001 0.0001 0.0002 ±0.17% 5871395
BENCH Summary
extreme-router - static.bench.js > matching [zero]
1.00x faster than hono-regex-match
1.02x faster than fast-path-parse[-runtime]
1.02x faster than fast-path-parse[-aot]
1.03x faster than fast-node-parse
1.04x faster than wired
1.47x faster than find-my-way
1.48x faster than matchit-wasm
4.15x faster than matchit-native
5.52x faster than hono-trie-match
fast-node-parse - static.bench.js > parse [zero]
1.02x faster than extreme-router
1.04x faster than hono-regex-match
1.04x faster than fast-path-parse[-aot]
1.05x faster than fast-path-parse[-runtime]
1.62x faster than find-my-way
3.20x faster than matchit-wasm
5.19x faster than matchit-native
5.85x faster than hono-trie-match
extreme-router - static.bench.js > lookup [zero]
1.04x faster than hono-regex-match
1.04x faster than fast-node-parse
2.07x faster than find-my-way
6.00x faster than hono-trie-match
NaNx faster than wired
8.00x faster than matchit-wasm
11.21x faster than matchit-native
hono-regex-match - static.bench.js > matching [zero]
1.00x faster than fast-path-parse[-runtime]
1.03x faster than fast-path-parse[-aot]
1.12x faster than fast-node-parse
1.34x faster than extreme-router
1.68x faster than matchit-wasm
1.88x faster than find-my-way
2.07x faster than wired
4.05x faster than matchit-native
6.25x faster than hono-trie-match
hono-regex-match - static.bench.js > parse [zero]
1.03x faster than fast-node-parse
1.03x faster than fast-path-parse[-runtime]
1.06x faster than fast-path-parse[-aot]
1.89x faster than find-my-way
2.82x faster than matchit-wasm
5.25x faster than matchit-native
6.01x faster than hono-trie-match
NaNx faster than extreme-router
fast-node-parse - static.bench.js > lookup [zero]
1.03x faster than hono-regex-match
2.28x faster than find-my-way
6.64x faster than hono-trie-match
NaNx faster than wired
NaNx faster than extreme-router
7.37x faster than matchit-wasm
10.25x faster than matchit-native
fast-node-parse - static.bench.js > matching [easy]
1.05x faster than extreme-router
1.29x faster than fast-path-parse[-aot]
1.30x faster than fast-path-parse[-runtime]
1.51x faster than wired
1.58x faster than matchit-wasm
2.12x faster than find-my-way
2.33x faster than hono-regex-match
3.44x faster than matchit-native
11.89x faster than hono-trie-match
fast-node-parse - static.bench.js > parse [easy]
1.07x faster than fast-path-parse[-runtime]
1.14x faster than fast-path-parse[-aot]
2.21x faster than find-my-way
2.27x faster than hono-regex-match
11.67x faster than hono-trie-match
NaNx faster than extreme-router
16.92x faster than matchit-wasm
18.46x faster than matchit-native
fast-node-parse - static.bench.js > lookup [easy]
2.51x faster than hono-regex-match
2.70x faster than find-my-way
12.46x faster than hono-trie-match
NaNx faster than wired
NaNx faster than extreme-router
22.86x faster than matchit-wasm
25.71x faster than matchit-native
wired - static.bench.js > matching [easy]
1.09x faster than fast-node-parse
1.32x faster than fast-path-parse[-aot]
1.43x faster than extreme-router
1.48x faster than fast-path-parse[-runtime]
1.72x faster than matchit-wasm
2.21x faster than hono-regex-match
2.23x faster than find-my-way
3.77x faster than matchit-native
8.65x faster than hono-trie-match
fast-node-parse - static.bench.js > parse [easy]
1.06x faster than fast-path-parse[-aot]
1.16x faster than fast-path-parse[-runtime]
2.29x faster than find-my-way
2.39x faster than hono-regex-match
9.02x faster than hono-trie-match
NaNx faster than extreme-router
19.14x faster than matchit-wasm
21.62x faster than matchit-native
fast-node-parse - static.bench.js > lookup [easy]
2.35x faster than hono-regex-match
2.71x faster than find-my-way
9.10x faster than hono-trie-match
NaNx faster than wired
NaNx faster than extreme-router
22.82x faster than matchit-wasm
26.51x faster than matchit-native
fast-node-parse - static.bench.js > matching [medium]
1.26x faster than extreme-router
1.58x faster than fast-path-parse[-aot]
1.84x faster than fast-path-parse[-runtime]
2.23x faster than hono-regex-match
2.31x faster than matchit-wasm
2.32x faster than wired
4.03x faster than matchit-native
4.74x faster than find-my-way
19.94x faster than hono-trie-match
fast-node-parse - static.bench.js > parse [medium]
1.45x faster than fast-path-parse[-aot]
1.79x faster than fast-path-parse[-runtime]
2.73x faster than hono-regex-match
4.36x faster than find-my-way
19.77x faster than hono-trie-match
NaNx faster than extreme-router
31.77x faster than matchit-wasm
32.65x faster than matchit-native
fast-node-parse - static.bench.js > lookup [medium]
2.64x faster than hono-regex-match
4.57x faster than find-my-way
18.83x faster than hono-trie-match
NaNx faster than wired
NaNx faster than extreme-router
34.58x faster than matchit-wasm
37.60x faster than matchit-native
PASS Waiting for file changes...
press h to show help, press q to quit
Reproduction
// eslint-disable-next-line @eslint-community/eslint-comments/disable-enable-pair
/* eslint-disable max-lines */
import FindMyWay from 'find-my-way';
import { RegExpRouter } from 'hono/router/reg-exp-router';
import { TrieRouter } from 'hono/router/trie-router';
import { CallbackNode, RootNode } from './nodes.js';
import fppMatchR from 'fast-path-parse/runtime/match';
import fppMatchA from 'fast-path-parse/aot/match';
import fppParseR from 'fast-path-parse/runtime/parse';
import fppParseA from 'fast-path-parse/aot/parse';
import { describe, bench } from 'vitest';
import { Wired } from './wired.js';
import Extreme, { param, wildcard } from 'extreme-router';
/**
* DO NOT TOUCH
* THIS IS TESTING CONSTANT
*/
/** @type {import('vitest').BenchOptions} */
const globalBenchConfig = {
iterations: 1_000_000,
warmupIterations: 1_000,
now: process.now
};
const modes = [
{
path: '/foo',
complexity: 'zero',
validate: '/foo'
},
{
path: '/foo/bar',
complexity: 'zero',
validate: '/foo/bar'
},
{
path: '/foo/:bar',
complexity: 'easy',
validate: '/foo/bar'
},
{
path: '/foo/*',
complexity: 'easy',
validate: '/foo/bar'
},
{
path: '/user/:id/edit/:page',
complexity: 'medium',
validate: '/user/1234/edit/weather'
}
];
for (const { path, complexity, validate } of modes) {
const root = new RootNode('/');
const foo = new CallbackNode(
'GET',
() => {
return 1 + (1 * 2) / 2 + 1;
},
'foo'
);
const fastPathParseMatchRuntime = fppMatchR(path);
const fastPathParseMatchAoT = fppMatchA(path);
const fastPathParseParseRuntime = fppParseR(path);
const fastPathParseParseAoT = fppParseA(path);
const wire = new Wired();
const extreme = new Extreme();
extreme.use(param).use(wildcard);
root.push(foo);
root.optimize(['method', 'callback']);
const route = FindMyWay({
defaultRoute() {
return false;
}
});
route.on('GET', path, () => () => {
return 1 + (1 * 2) / 2 + 1;
});
const honoRouteRegEx = new RegExpRouter();
honoRouteRegEx.add('GET', path, () => {
return 1 + (1 * 2) / 2 + 1;
});
const honoTrieRoute = new TrieRouter();
honoTrieRoute.add('GET', path, () => {
return 1 + (1 * 2) / 2 + 1;
});
wire.add('GET', path, () => {
return 1 + (1 * 2) / 2 + 1;
});
extreme.register('/foo').handler = () => {
return 1 + (1 * 2) / 2 + 1;
};
describe(`matching [${complexity}]`, () => {
bench(
'fast-node-parse',
() => {
return root.match(validate);
},
globalBenchConfig
);
bench(
'fast-path-parse[-runtime]',
() => {
return fastPathParseMatchRuntime(validate);
},
globalBenchConfig
);
bench(
'fast-path-parse[-aot]',
() => {
return fastPathParseMatchAoT(validate);
},
globalBenchConfig
);
bench(
'hono-regex-match',
() => {
return honoRouteRegEx.match('GET', validate);
},
globalBenchConfig
);
bench(
'hono-trie-match',
() => {
return honoTrieRoute.match('GET', validate);
},
globalBenchConfig
);
bench(
'find-my-way',
() => {
return route.find('GET', validate);
},
globalBenchConfig
);
bench(
'wired',
() => {
return wire.has('GET', validate);
},
globalBenchConfig
);
bench(
'extreme-router',
() => {
return extreme.match(validate);
},
globalBenchConfig
);
});
// Suite
describe(`parse [${complexity}]`, () => {
bench(
'fast-node-parse',
() => {
return root.parse(validate);
},
globalBenchConfig
);
bench(
'fast-path-parse[-runtime]',
() => {
return fastPathParseParseRuntime(validate);
},
globalBenchConfig
);
bench(
'fast-path-parse[-aot]',
() => {
return fastPathParseParseAoT(validate);
},
globalBenchConfig
);
bench(
'hono-regex-match',
() => {
const [res] = honoRouteRegEx.match('GET', validate);
const params = res[0][1];
return params;
},
globalBenchConfig
);
bench(
'hono-trie-match',
() => {
const [res] = honoTrieRoute.match('GET', validate);
const params = res[0][1];
return params;
},
globalBenchConfig
);
bench(
'find-my-way',
() => {
return route.find('GET', validate);
},
globalBenchConfig
);
bench(
'extreme-router',
() => {
return extreme.match(validate).params;
},
globalBenchConfig
);
});
// Suite
describe(`lookup [${complexity}]`, () => {
bench(
'fast-node-parse',
() => {
return root.lookup(validate);
},
globalBenchConfig
);
bench(
'hono-regex-match',
() => {
const [res] = honoRouteRegEx.match('GET', validate);
const [callback, params] = res[0];
return callback(params);
},
globalBenchConfig
);
bench(
'hono-trie-match',
() => {
const [res] = honoTrieRoute.match('GET', validate);
const [callback, params] = res[0];
return callback(params);
},
globalBenchConfig
);
bench(
'find-my-way',
() => {
return route.lookup({ method: 'GET', url: validate }, {});
},
globalBenchConfig
);
bench(
'wired',
() => {
return wire.lookup('GET', validate)();
},
globalBenchConfig
);
bench(
'extreme-router',
() => {
const { handler, params } = extreme.match(validate);
return handler(params);
},
globalBenchConfig
);
});
}
System Info
System:
OS: macOS 15.7
CPU: (10) arm64 Apple M1 Max
Memory: 2.81 GB / 64.00 GB
Shell: 4.0.8 - /opt/homebrew/bin/fish
Binaries:
Node: 22.19.0 - ~/.local/state/fnm_multishells/49780_1758898988127/bin/node
Yarn: 1.22.22 - /opt/homebrew/bin/yarn
npm: 10.9.3 - ~/.local/state/fnm_multishells/49780_1758898988127/bin/npm
pnpm: 10.17.1 - /opt/homebrew/bin/pnpm
bun: 1.2.22 - /opt/homebrew/bin/bun
Watchman: 2025.09.15.00 - /opt/homebrew/bin/watchman
Browsers:
Safari: 26.0.1
npmPackages:
vitest: ^3.2.4 => 3.2.4
Used Package Manager
npm
Validations
- [x] Follow our Code of Conduct
- [x] Read the Contributing Guidelines.
- [x] Read the docs.
- [x] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- [x] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
- [x] The provided reproduction is a minimal reproducible example of the bug.
This happened to me when exceptions were being thrown inside the benchmarks. Unfortunately, Vitest didn’t report them. After fixing the errors, the speedup displayed correctly.
As a workaround, you can use { throws: true } option for bench while you author the benchmark:
bench("my test", () => {
[].sort()
throw new Error("bad")
}, { throws: true })
But yes, it took a while to track this down so I would have preferred throws: true as the default.
https://github.com/vitest-dev/vitest/issues/8628#issuecomment-3603691588
This solution worked, thank you @pcattori
@dalisoft glad it helped!
That said, I feel like this issue should stay open since my comment is really a workaround. The core is still exists: users get no feedback about what is causing the NaNx faster than in normal test development.