Tinybench runs the test cases in parallel which is bad for tests that access resources like files

[BioPhoton]

Problem: I wrote a benchmark to compare file crawling and get quite different results compared to e.g. benchmark. With benchmark I'm able to run tests in sequence so they can all after another access the same resources. In this way the results are reliable.

With tinybench I throw out of memory or other odd errors due to parallel processing.

Suggested solution:

Implement a way to run all test cases in sequence.

Links:

• https://github.com/bestiejs/benchmark.js/issues/176#issuecomment-1815825325

[Aslemammad]

Sure, @sirenkovladd might be interested?

[peterHakio]

Also relevant in context of vitest https://github.com/vitest-dev/vitest/issues/5412

[Aslemammad]

Tasks are already running sequentially. https://github.com/tinylibs/tinybench/blob/main/test/sequential.test.ts

So let me know what is the specific issue! As far as I know, vitest does not use bench.run from tinybench and it implements its own way.

[Aslemammad]

@BioPhoton Do you have any reproduciton?

[BioPhoton]

I have the following suite:

// eslint-ignore-next-line import/no-named-as-default-member
import fastGlob from 'fast-glob';
import { glob } from 'glob';
import { globby } from 'globby';
import { join } from 'node:path';
import yargs from 'yargs';

const cli = yargs(process.argv).options({
  pattern: {
    type: 'array',
    string: true,
    default: [
      join(process.cwd(), 'node_modules/**/*.md'),
    ],
  },
  outputDir: {
    type: 'string',
  },
  logs: {
    type: 'boolean',
    default: true,
  },
});

// eslint-disable-next-line n/no-sync
const { pattern, logs } = cli.parseSync();

if (logs) {
  // eslint-disable-next-line no-console
  console.log('You can adjust the test with the following arguments:');
  // eslint-disable-next-line no-console
  console.log(
    `pattern      glob pattern of test      --pattern=${pattern.toString()}`,
  );
}

const fastGlobName = 'fast-glob';
const globName = 'glob';
const globbyName = 'globby';

// ==================

const suiteConfig = {
  suiteName: 'glob-matching',
  cases: [
    [fastGlobName, callAndValidate(fastGlob.async, pattern, fastGlobName)],
    [globName, callAndValidate(glob, pattern, globName)],
    [globbyName, callAndValidate(globby, pattern, globbyName)],
  ]
};
export default suiteConfig;

// ==============================================================
const logged: Record<string, boolean> = {};
function callAndValidate<T = string | string[]>(
  fn: (patterns: T) => Promise<unknown[]>,
  globPatterns: T,
  fnName: string,
) {
  return async () => {
    const result = await fn(globPatterns);
    if (result.length === 0) {
       throw new Error(`Result length is ${result.length}. This test is not meaningful`);
    } else {
      if (!logged[fnName]) {
        logged[fnName] = true;
        console.log(
          `${fnName} found ${result.length} files for pattern ${pattern.join(
            ', ',
          )}`,
        );
      }
    }
  };
}

Somehow it produces different test results than expected.

[Aslemammad]

Ok, I'll check this out soon.

[jerome-benoit]

I have the following suite:

I do not see where tinybench API is used ... are you sure the bug is filled against the right project?

[jerome-benoit]

No activity since 3 weeks.