bench_func returning None for all but the first benchmark

[axman6]

I'm trying to write a benchmark suite for several functions and use BenchmarkSuite to write the results to a json file. I have the following code (irrelevant parts removed):

import pyperf
import decimation_accel as dec
import decimation_accel_opt as opt


results_file = 'decimation.json'

runner = pyperf.Runner()


def main() -> None:

    small = alloc_args(1024)
    small10 = alloc_args(1024, rate=10)
    small100 = alloc_args(1024, rate=100)
    medium = alloc_args(1024 * 10)
    benchmarks: List[pyperf.Benchmark] = []


    benchmarks.append(runner.bench_func('dec.decimate_minmax(small) all', dec.minmax, *small))
    benchmarks.append(runner.bench_func('opt.decimate_minmax(small) all', opt.minmax, *small))
    benchmarks.append(runner.bench_func('dec.decimate_minmax(small10) all', dec.minmax, *small10))
    benchmarks.append(runner.bench_func('opt.decimate_minmax(small10) all', opt.minmax, *small10))
    benchmarks.append(runner.bench_func('dec.decimate_minmax(small100) all', dec.minmax, *small100))
    benchmarks.append(runner.bench_func('opt.decimate_minmax(small100) all', opt.minmax, *small100))
    benchmarks.append(runner.bench_func('dec.decimate_minmax(medium) all', dec.minmax, *medium))
    benchmarks.append(runner.bench_func('opt.decimate_minmax(medium) all', opt.minmax, *medium))

    print(benchmarks)

    suite = pyperf.BenchmarkSuite(benchmarks)  # filename='decimation.json')

    suite.dump(results_file)


if __name__ == "__main__":
    main()

However if I run this file, I get the following output:

[<Benchmark 'dec.decimate_minmax(small) all' with 1 runs>, None, None, None, None, None, None, None]
Traceback (most recent call last):
  File "benchmarks.py", line 60, in <module>
    main()
  File "benchmarks.py", line 54, in main
    suite = pyperf.BenchmarkSuite(benchmarks)  # filename='decimation.json')
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_bench.py", line 639, in __init__
    self.add_benchmark(benchmark)
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_bench.py", line 687, in add_benchmark
    name = benchmark.get_name()
AttributeError: 'NoneType' object has no attribute 'get_name'
Traceback (most recent call last):
  File "benchmarks.py", line 60, in <module>
    main()
  File "benchmarks.py", line 39, in main
    benchmarks.append(runner.bench_func('dec.decimate_minmax(small) all', dec.minmax, *small))
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_runner.py", line 537, in bench_func
    result = self._main(task)
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_runner.py", line 460, in _main
    bench = self._manager()
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_runner.py", line 673, in _manager
    bench = Manager(self).create_bench()
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_manager.py", line 232, in create_bench
    worker_bench, run = self.create_worker_bench()
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_manager.py", line 131, in create_worker_bench
    suite = self.create_suite()
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_manager.py", line 121, in create_suite
    suite = self.spawn_worker(self.calibrate_loops, 0)
  File "<REMOVED>/venv-local/lib/python3.8/site-packages/pyperf/_manager.py", line 107, in spawn_worker
    raise RuntimeError("%s failed with exit code %s"
RuntimeError: <REMOVED>/venv-local/bin/python failed with exit code 1

If I don't put the results in the list, then they all appear to run fine, but then I have no way to collect the results.

[lrntct]

I just encountered the exact same issue and I'd like to know if there is something I do wrong or a workaround. I use pyperf 2.9.0

[maurycy]

A simpler way to reproduce:

import pyperf


def fake_bm():
    pass


runner = pyperf.Runner()
benchmarks = []
benchmarks.append(runner.bench_func("first_run", fake_bm))
benchmarks.append(runner.bench_func("second_run", fake_bm))

suite = pyperf.BenchmarkSuite(benchmarks)

[vstinner]

pyperf has an unusual design, it spawns worker processes which re-executes the script. Only the main process gets Benchmark objects when calling bench_func(). Worker processes get None instead.

One workaround would be to add a Runner.is_worker() method to check if the process is the main process or a worker process.

[amritamishra01]

Hi! I’d like to work on this issue.

From debugging the behavior, the problem is that bench_func() is executed both in the main process and in the worker processes. Only the main process should produce a Benchmark object, while worker processes correctly return None—but these None values end up in user code and break BenchmarkSuite.

Based on @vstinner’s suggestion, my plan is:

Add a Runner.is_worker() helper to reliably detect worker processes

Ensure bench_func() only returns a Benchmark object in the main process

Make BenchmarkSuite ignore any None values for safety

Add a small test validating that multiple bench_func() calls from the main process return valid Benchmark objects

If this approach looks good, I’ll start working on a PR.

Thanks!

[vstinner]

I suggest by starting only with adding a is_worker() method and nothing else.