Runner.bench_func can't run different module's function?

[mlouielu]

I'm trying to run Runner.bench_func inside a package like this:

└── format
    ├── benchmarks
    │   ├── bm_format.py
    │   ├── bm_fstring.py
    │   ├── bm_percentage.py
    │   └── __init__.py
    ├── __init__.py
    └── run.py

Inside format/run.py the code will run benchmarks' function like this:

import perf
from .benchmarks import BENCHS


def run():
    runner = perf.Runner()
    for bm_name, bm_func in BENCHS:
        runner.bench_func(bm_name, bm_func)


if __name__ == '__main__':
    run()

When running this inside cli, it will raise RuntimeError and ModuleNotFoundError:

➜  python-string-format python -m format.run
Traceback (most recent call last):
  File "/tmp/python-string-format/format/run.py", line 2, in <module>
    from .benchmarks import BENCHS
ModuleNotFoundError: No module named '__main__.benchmarks'; '__main__' is not a package
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/python-string-format/format/run.py", line 12, in <module>
    run()
  File "/tmp/python-string-format/format/run.py", line 8, in run
    runner.bench_func(bm_name, bm_func)
  File "/usr/lib/python3.6/site-packages/perf/_runner.py", line 594, in bench_func
    return self._main(name, sample_func, inner_loops, metadata)
  File "/usr/lib/python3.6/site-packages/perf/_runner.py", line 508, in _main
    bench = self._master()
  File "/usr/lib/python3.6/site-packages/perf/_runner.py", line 768, in _master
    bench = self._spawn_workers()
  File "/usr/lib/python3.6/site-packages/perf/_runner.py", line 724, in _spawn_workers
    suite = self._spawn_worker(calibrate)
  File "/usr/lib/python3.6/site-packages/perf/_runner.py", line 665, in _spawn_worker
    % (cmd[0], exitcode))
RuntimeError: /usr/bin/python failed with exit code 1

Do I did wrong something, or the Runner.bench_func can only run inside its module?

Thanks.

[mlouielu]

Hmm, I can run via $ python -m format.run, but changing relative import from .bench ... to from bench ..., then run via $ python format will work..

[vstinner]

from .benchmarks import BENCHS

Relative Python imports depends how you run you code. sys.argv[0] can be different than your command line. The Runner class has a program_args argument to workaround the sys.argv[0] issue: http://perf.readthedocs.io/en/latest/api.html#runner-class

Why not using absolute imports?

See also how http://github.com/python/performance handles this issue.

Reminder: perf spawns worker child process, so it has to build a command line to run them.

[vstinner]

How can we enhance the documentation to handle this issue?

[mlouielu]

@haypo after using absolute imports inside the project, I still can do like this python -m formats

https://github.com/mlouielu/python-string-format-microperformance

[vstinner]

@haypo after using absolute imports inside the project, I still can do like this python -m formats

That's why I consider that it's more a documentation issue than a bug.

Can I close this bug report? Or do you want to write a PR for the doc?

[mlouielu]

Ah.. That is a typo, I * can not * do like this python -m formats, it get me this output:

Traceback (most recent call last):
  File "/Users/louielu/dev/python-string-format-microperformance/formats/__main__.py", line 1, in <module>
    from formats.run import run
ModuleNotFoundError: No module named 'formats'
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/louielu/dev/python-string-format-microperformance/formats/__main__.py", line 3, in <module>
    run()
  File "/Users/louielu/dev/python-string-format-microperformance/formats/run.py", line 8, in run
    runner.bench_func(bm_name, bm_func)
  File "/usr/local/lib/python3.6/site-packages/perf/_runner.py", line 485, in bench_func
    return self._main(task)
  File "/usr/local/lib/python3.6/site-packages/perf/_runner.py", line 420, in _main
    bench = self._master()
  File "/usr/local/lib/python3.6/site-packages/perf/_runner.py", line 543, in _master
    bench = Master(self).create_bench()
  File "/usr/local/lib/python3.6/site-packages/perf/_master.py", line 221, in create_bench
    worker_bench, run = self.create_worker_bench()
  File "/usr/local/lib/python3.6/site-packages/perf/_master.py", line 120, in create_worker_bench
    suite = self.create_suite()
  File "/usr/local/lib/python3.6/site-packages/perf/_master.py", line 110, in create_suite
    suite = self.spawn_worker(self.calibrate_loops, 0)
  File "/usr/local/lib/python3.6/site-packages/perf/_master.py", line 97, in spawn_worker
    % (cmd[0], exitcode))
RuntimeError: /usr/local/Cellar/python3/3.6.1/bin/python3.6 failed with exit code 1

[boustrophedon]

I am also unable to use absolute imports when attempting to run benches with python -m with the following directory structure:

mypkg/
mypkg/__init__.py
mypkg/myfunc.py
mypkg/bench/
mypkg/bench/bench_myfunc.py

The contents of bench_myfunc.py is:

from mypackage.myfunc import myfunc

if __name__ == '__main__':
  runner = perf.Runner()
  runner.bench_func("myfunc_foo", myfunc)

and when I run python -m mypkg.bench.bench_myfunc I get the same error as above:

Traceback (most recent call last):
  File "/tmp/perf_repro/mypkg/bench/bench_myfunc.py", line 1, in <module>
    from mypkg.myfunc import myfunc
ModuleNotFoundError: No module named 'mypkg'
Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/perf_repro/mypkg/bench/bench_myfunc.py", line 6, in <module>
    runner.bench_func("bench_myfunc", myfunc)
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_runner.py", line 493, in bench_func
    return self._main(task)
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_runner.py", line 428, in _main
    bench = self._master()
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_runner.py", line 551, in _master
    bench = Master(self).create_bench()
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_master.py", line 221, in create_bench
    worker_bench, run = self.create_worker_bench()
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_master.py", line 120, in create_worker_bench
    suite = self.create_suite()
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_master.py", line 110, in create_suite
    suite = self.spawn_worker(self.calibrate_loops, 0)
  File "/home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/lib/python3.7/site-packages/perf/_master.py", line 97, in spawn_worker
    % (cmd[0], exitcode))
RuntimeError: /home/hcs/.local/share/virtualenvs/perf_repro-bPiNGceB/bin/python failed with exit code 1

I am using python 3.7 and perf 1.5.1 inside a virtualenv created by pipenv.

I have attached a tar containing the repro case, with an additional file that shows that importing is working: python -m mypkg.bench.import_works perf_absolute_import_repro.tar.gz