Benchmark discovery attempts to instantiate everything it finds

[astrojuanlu]

As part of the SciPy US 2020 sprints, after not touching our asv benchmarks for a while, @samtx tried to update them and found that the discovery step tries to instantiate all the classes that it finds, even ones that are not benchmarks. As a result, we were seeing weird errors like this one:

Traceback (most recent call last):
  File "/home/juanlu/.pyenv/versions/3.8.3/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/home/juanlu/.pyenv/versions/3.8.3/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/home/juanlu/.pyenv/versions/3.8.3/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/juanlu/.pyenv/versions/3.8.3/envs/poliastro38_benchmarks/lib/python3.8/site-packages/asv/benchmark.py", line 3, in <module>
    """\
  File "/home/juanlu/.pyenv/versions/3.8.3/envs/poliastro38_benchmarks/lib/python3.8/site-packages/asv/benchmark.py", line 1308, in main
    commands[mode](args)
  File "/home/juanlu/.pyenv/versions/3.8.3/envs/poliastro38_benchmarks/lib/python3.8/site-packages/asv/benchmark.py", line 1004, in main_discover
    list_benchmarks(benchmark_dir, fp)
  File "/home/juanlu/.pyenv/versions/3.8.3/envs/poliastro38_benchmarks/lib/python3.8/site-packages/asv/benchmark.py", line 989, in list_benchmarks
    for benchmark in disc_benchmarks(root):
  File "/home/juanlu/.pyenv/versions/3.8.3/envs/poliastro38_benchmarks/lib/python3.8/site-packages/asv/benchmark.py", line 896, in disc_benchmarks
    benchmark = _get_benchmark(name, module, module_attr,
  File "/home/juanlu/.pyenv/versions/3.8.3/envs/poliastro38_benchmarks/lib/python3.8/site-packages/asv/benchmark.py", line 838, in _get_benchmark
    instance = klass()
TypeError: __init__() missing 2 required positional arguments: 'state' and 'epoch'

by launching python -m pdb /path/to/asv/benchmark.py discover /path/to/benchmarks/benchmarks temp_result.json, we saw that klass is <class 'poliastro.twobody.orbit.Orbit'>, a class that is imported at the top of the benchmark.

I tried to look for a mention of this in the issue tracked or the documentation, but found nothing. Also, the examples in https://asv.readthedocs.io/en/stable/writing_benchmarks.html only use the standard library, so they don't have this problem. And finally, I saw that @sklam had the same issue with the numba benchmarks at https://github.com/numba/numba-benchmark/pull/6.

I am tempted to say that this behavior should be considered a bug, but I acknowledge that it might be difficult to fix. However, perhaps a broad try ... except that discards error in the disc_benchmarks function should be enough?

https://github.com/airspeed-velocity/asv/blob/160f21186e86698b5e9256587f3ccd36f7f44f37/asv/benchmark.py#L931

And, regardless of potential changes that could be made for asv, what's the current recommendation for dealing with imports?