Seeing a lot of "cannot find thread state, sample will be thrown away" warning messages

[rappdw]

Running against latest master (ccae9f7d9148634fd3366e504e97deb4b7e29eb7)

I consistently see:

WARNING: cannot find thread state, sample will be thrown away WARNING: get_stack_trace, current is NULL WARNING: canceled buffer, no stack trace was written

This occurs on both OSX and Linux (debian)

[planrich]

That is a real problem. That means that if the signal occurs it cannot find the python PyThreadState to sample the stack. Does this only happen for a specific version of CPython?

I have now restricted the signal to occur if Py_IsInitialized() returns != 0 (in 2d8e8ff10) this should remove all warnings when the interpreter is teared down.

[ghackebeil]

I'm seeing this as well with Python 2.7.13 built from source on a Linux machine. I'm using vmprof master (7dba52895e85eb860032f212239a220912f8465f) installed using pip.

I've determined that these warnings appear solely due to the import of NumPy (1.13.0 installed via pip). I've even narrowed this down to being triggered by the import of any one of the C extension modules in numpy/core (e.g., numpy/core/umath.so or numpy/core/multiarray.so). To reproduce this, all you need to do is cd into site-packages/python2.7/numpy/core, create a file junk.py with the line import multiarray, and run python -m vmprof junk.py.

[ghackebeil]

Update:

NumPy master (tested on numpy/numpy@03283c289f6105da70e64948b91efe96700d974e, installed via pip) no longer triggers these warnings. Maybe this was an issue with NumPy that has been fixed.

[lopuhin]

I'm also getting this warnings with Python 3.6.2 on Ubuntu built from source, but I was unable to reduce the error to just importing some libraries.

[elistevens]

I also see similar messages:

WARNING: cannot find thread state (for thread id 140324874270464), sample will be thrown away
WARNING: get_stack_trace, current is NULL
WARNING: canceled buffer, no stack trace was written
WARNING: get_stack_trace, frame is NULL
WARNING: canceled buffer, no stack trace was written
...etc...

I'm using numpy 1.14.0, which includes the commit referenced above. I'm also using PyTorch under python 3, Ubuntu 16.04. The program I'm using uses multiprocessing via the PyTorch DataLoader class, though I haven't done any work to minimize an example.

Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux

[njsmith]

If you're seeing this with numpy, then I suspect the reason is that you're using np.dot or np.linalg or similar, so it's calling into an optimized BLAS/LAPACK library like OpenBLAS or MKL. And then that library is starting up multiple threads internally to farm out the work over multiple cores. Vmprof doesn't know what to do with these threads, since they're created independently of Python and aren't running Python code. And unfortunately, there's really no way for vmprof (or any profiler) to figure out that when it sees this thread over here using CPU, then it should count that against the np.dot call that's happening over in another thread somewhere else :-(.

I've found that running my script with the environment variable OMP_NUM_THREADS=1 can help – this disables threading in most BLAS libraries. Of course, now you're running with threading disabled, so you need to keep that in mind when interpreting your profiling data – it won't be as representative of how you normally run your script when you're not profiling. But the data can still be very useful.

[antocuni]

@njsmith would you consider adding an API to integrate numpy with profilers? Like some sort of callback which numpy invokes when creates a new thread, e.g.:

# in vmprof
import numpy

def on_thread_created(thread_id, description):
    my_global_registry[thread_id] = description # something like 'np.dot'
numpy.hooks.on_thread_created = on_thread_created
numpy.hooks.on_thread_destroyed = ...

[njsmith]

@antocuni numpy doesn't create threads. The threads I'm talking about are created by libraries like OpenBLAS, or Intel's MKL. So I'm afraid you're asking the wrong person :-).

[elistevens]

Would it be possible to only emit those particular warnings once? Or have a CLI arg to skip them entirely?

[gjcarneiro]

I am also seeing these warnings with our private C++ extension module (cofferdb) that reads data from network in a separate non-python thread.

I would rather just completely ignore unknown threads. This is a Python profiler, profiling non-Python threads should be out of scope of vmprof, IMHO.

[davit555]

Same for PyPy 6.0.0