Promise support

[sindresorhus]

Would be much nicer to be able to return a promise in a suite than having to manually call deferred.resolve().

This is what we currently have to do:

suite
	.add('baseline', {
		defer: true,
		fn(deferred) {
			const queue = new PQueue();
			for (let i = 0; i < 100; i++) {
				queue.add(() => Promise.resolve());
			}
			queue.onEmpty().then(() => deferred.resolve());
		}
	})

This is what I would like:

suite
	.add('baseline', {
		defer: true,
		fn() {
			const queue = new PQueue();
			for (let i = 0; i < 100; i++) {
				queue.add(() => Promise.resolve());
			}
			return queue.onEmpty();
		}
	})

In the next major version, I would also remove the async and defer option and just be async if the bench function returns a Promise, otherwise synchronous.

---

// @jdalton @floatdrop

---

Context: https://github.com/sindresorhus/p-queue/pull/4/files#diff-bf2f25928a0b46d29e7743d11b05f999R8

[sindresorhus]

@garkin I mention that in the above description.

[ghost]

I'll try to create a pull request for this. I really need this feature in a high-level benchmarking library I'm creating.

[prantlf]

Recognising Promise as the callback result type should be straightforward. Dropping the defer option will need some more work to remove the need to know about the asynchronous behaviour before running the benchmark (isAsync method, for example) by handling all callbacks as "potentially" asynchronous, for example. It would be an "insidious" breaking change targetting people, who returned values from their callbacks "at whim" ;-)

[kevlened]

For anyone looking for a quick copy/paste fix:

function p(fn) {
  return {
    defer: true,
    async fn(deferred) {
      await fn();
      deferred.resolve();
    }
  }
}

Usage:

const wait = ms => new Promise(res => setTimeout(res, ms));

new Suite()
  .add('An async perf test', p(async () => {
    await wait(1000);
  }))
  .on('cycle', ev => console.log(String(ev.target)))
  .run();

[Uzlopak]

I forked benchmark.js and i would actually say that determining If the Return value is a Promise is not reliable imho. But we can determine if the function is an async function and then wrap defer object around it.

[jdmarshall]

Am I misunderstanding the problem here or can this be solved by using async: true?

[Uzlopak]

Async means that the benchmarking is delayed by 0.05 seconds or so. It does not mean, that promises work or so. You have some callback/Promise resolve logic possible if you use defer: true and in the benchmark you resolve the deferred object.

You can checkout my fork callled tinybench. It has async functionality, but it is painfully slower and gives not correct benchmark results for async.

[Uzlopak]

tinybench is working flawless now.

[jdmarshall]

Knock knock.

[anonghuser]

Here is a small subclass that auto deferredifies async function benchmarks:

AsyncSuite.AsyncFunction = (async function() {}).constructor
Object.setPrototypeOf(AsyncSuite.prototype, Benchmark.Suite.prototype)
function AsyncSuite() {
    'use strict'
    const suite = (!new.target && this instanceof AsyncSuite) ? Benchmark.Suite.apply(this, arguments) : Reflect.construct(Benchmark.Suite, arguments, new.target || AsyncSuite)
    suite.promise = new Promise(resolve => {
        suite.on('add', e => {
            const benchmark = e.target
            if (!benchmark.defer && (!suite.options.checkAsync || benchmark.asyncFn || benchmark.fn instanceof AsyncSuite.AsyncFunction)) {
                benchmark.asyncFn = benchmark.options.asyncFn = benchmark.asyncFn || benchmark.fn
                benchmark.defer = benchmark.options.defer = true
                benchmark.fn = benchmark.options.fn = function(deferred) {
                    Promise.resolve(deferred.benchmark.asyncFn()).then(() => deferred.resolve(), e => {
                        deferred.benchmark.error = e
                        deferred.resolve()
                    })
                }
            }
        })
        suite.on('complete', function() {
            resolve(this)
        })
    })
    return suite
}

Adds a promise property that resolves when onComplete fires.

By default it wraps all functions that are not already defer: true which should be ok for most people.

For more control, i.e. if you want to have functions that return promises but are not waited or to avoid the overhead of defer: true for synchronous code, the checkAsync option can limit it to only wrap async functions or ones that use asyncFn option key instead of the regular fn. The first check is based on instanceof and not toString or consructor.name so probably will not work cross-realm.

Use in the obvious way:

const suite = new AsyncSuite(/*{checkAsync: true}*/)
suite.add(async function () {
    // see how it's very slow, properly waiting the timer
    return new Promise(resolve=>setTimeout(resolve, 100))
})
suite.add(async function () {
    // this one waits for a no-op so should be very fast, yet slower than sync version because of deferred overhead
})
suite.add(function () {
    // note the lack of async keyword before the function.
    // won't be wrapped if `checkAsync: true` option is set so should be the fastest, but even if it returned a promise it won't be waited.
    // same as the previous one without the option, if it returned a promise it would be properly waited.
})
console.log((await suite.run().promise).map('hz')) // don't feel obligated to golf it to one-line in your own code :p

Naturally, all Benchmark.Suite APIs (options/events/methods/properties) remain supported as well.

[Uzlopak]

@anonghuser

I dont see that you set delay to 0 in your snippet

[anonghuser]

@Uzlopak The delay option is applied before the whole benchmark, not between its iterations, and is not included in the measured timings so does not affect the result. There is no need to change anything about it.