evanwashere
Feature: before / after bench hooks?
Sometimes when we bench some code, we need to prepare the environment. Think of a scenario like when we try to compare the performance of two different database indexes. The baseline can be a query without index, and two bench statements both with their own unique index. Pseudo:
group('select 10 users', () => {
baseline('no index', async () => {
await client.query('select * from users order by created_at asc, id asc limit 10');
});
bench('with created_at index', async() => {
await client.query(`select * from users order by created_at asc, id asc limit 10`);
}, {
before: async () => dropIndexes().then(() => createFirstIndex()),
after: async () => dropIndexes(),
});
bench('with created_at and id index', async() => {
await client.query(`select * from users order by created_at asc, id asc limit 10`);
}, {
before: async () => dropIndexes().then(() => createAnotherIndex()),
after: async () => dropIndexes(),
});
});
I think it would be useful to have it both on the group level, as well as on the bench level. The group could in this case for example be used to connect/disconnect to the database, or to support multiple query tests for a single created index, where another group would test another index.
I think a pretty solution - better than the above - could be to add a prepare handler that runs in order of the benchmarks, but only once. Something like:
group('select 10 users', () => {
prepare('clean', async () => clean());
baseline('no index', async () => client.query(...));
prepare('create index 1', async () => clean() && createIndex1());
bench('select 10 users using index 1', async () => client.query(...));
prepare('create index 2', async () => clean() && createIndex2());
bench('select 10 users using index 2', async () => client.query(...));
prepare('cleanup', async () => clean());
});
It's something that I am aware of and trying to solve with 1.0.0 release to simplify everything I'm leaning towards generators and for loops design
bench(function *() {
// prep work
const index = ...;
yield () => {
// benchmark code
}
// cleanup work
index.deinit();
})
// and
bench(ctx => {
// prep
for (const _ of ctx) {
// benchmarked code
}
// cleanup
})
@evanwashere I'd like to add a small +1 in favour of the generator design as it looks like it would really nicely separate out the prep and cleanup from the timing of the actual code being benchmarked.
Plus supporting async generators would allow nice async setup/teardown in combination with synchronous benchmarked code:
bench(async function *() {
// prep work
const index = await ...;
yield () => {
// benchmark code
}
// cleanup work
await index.deinit();
})
