asv

asv copied to clipboard

I'm trying to benchmark a project, but asv seems to be struggling to discover tests. The output is

$ asv run -v
· Running '/usr/local/bin/git rev-parse master^{commit}'
  OUTPUT -------->
  647aa86cac45ba6244a23cb66a1dbb418235dbdd
· Creating environments
·· Running '/Users/dstansby/github/pfsspy/.asv/env/063e9b2d74f371a5aa0c00791c9eb7cb/bin/python -c pass'
· Discovering benchmarks
·· Running '/usr/local/bin/git rev-parse master^{commit}'
   OUTPUT -------->
   647aa86cac45ba6244a23cb66a1dbb418235dbdd
·· Running '/Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py discover /Users/dstansby/github/pfsspy/benchmarks /var/folders/6y/j8hmrrxs1vz3bmtvzvgln5zh0000gn/T/tmp2g1coiwj/result.json' in conda-py3.7-numpy-sunpy
·· Running '/Users/dstansby/github/pfsspy/.asv/env/063e9b2d74f371a5aa0c00791c9eb7cb/bin/python /Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py discover /Users/dstansby/github/pfsspy/benchmarks /var/folders/6y/j8hmrrxs1vz3bmtvzvgln5zh0000gn/T/tmp2g1coiwj/result.json'

and it hangs on the last line. I think the issue is in my benchmarks.py. If I comment out my package import, it works fine, but I need to import my package in benchmarks.py to benchmark it!

Sorry that this isn't a very helpful bug report; I'm not sure how I can further diagnose the problem however, so if anyone has any suggestions that would be great!

Edit: more info.

Running the line it seems to be getting stuck on, works fine:

$ pythonw /Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py discover /Users/dstansby/github/pfsspy/benchmarks result.json

So I'm not sure what asv run is getting stuck on, or what is different with the call asv run makes and the terminal command above.

Can you run the precise command line it shows i.e. the last /Users/dstansby/github/pfsspy/.asv/env/063e9b2d74f371a5aa0c00791c9eb7cb/bin/python /Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py discover /Users/dstansby/github/pfsspy/benchmarks /var/folders/6y/j8hmrrxs1vz3bmtvzvgln5zh0000gn/T/tmp2g1coiwj/result.json

Does that get stuck? Does it get stuck when the command is run from Python via subprocess.check_output --- or via asv.util.check_output?

No reply, closing.

Sorry for the long time gone, I'm trying this again with the same project and asv 0.4.1, and I'm still having the same problem. In answer to your questions:

/Users/dstansby/github/pfsspy/.asv/env/a80b4bcff004e7d21aafa78a5932418b/bin/python /Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py discover /Users/dstansby/github/pfsspy/benchmarks .result.json

Works fine (I've replaced the /var/folders... by a local directory otherwise it complains the temporary directory isn't available)

import subprocess
subprocess.check_output([
    '/Users/dstansby/github/pfsspy/.asv/env/a80b4bcff004e7d21aafa78a5932418b/bin/python',
    '/Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py',
    'discover',
    '/Users/dstansby/github/pfsspy/benchmarks',
    '.result.json'])

also works fine, but

import asv
asv.util.check_output([
    '/Users/dstansby/github/pfsspy/.asv/env/a80b4bcff004e7d21aafa78a5932418b/bin/python',
    '/Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/benchmark.py',
    'discover',
    '/Users/dstansby/github/pfsspy/benchmarks',
    '.result.json'])

hangs at

^CTraceback (most recent call last):
  File "test.py", line 7, in <module>
    '.result.json'])
  File "/Users/dstansby/miniconda3/lib/python3.7/site-packages/asv/util.py", line 607, in check_output
    list(fds.keys()), [], [], timeout)

Can you run benchmark suites from other projects / the minimal quickstart one? Do they hang? If yes, try commenting out parts from your benchmark suite to find out what causes it, to narrow things down?

Yep, other projects and the minimal quickstart one works fine. The issue on my project is when I import the project package in benchmarks.py - will try and isolate if there's something in my project __init__.py that's causing the hang.

@dstansby i may be experiencing similar problems, did you make some progress ?

I did not make any progress, sorry 😢 . I'm going to close this as I've now lost the original case where I had the issue. I think if someone else has this problem they should open a new issue, hopefully with more info than I could provide!