benfred
Include libpython symbols in profile info
AFAICT, when profiling native extensions, py-spy includes symbols from third-party shared objects, but not from libpython (PyFoo_...) itself. This can be desirable sometimes, and was provided e.g. by the (now somewhat dormant?) vmprof profiler. (An example application is profiling an extension that calls back into libpython and where these calls are suspected to take a significant amount of time.)
Which pythons symbols are you interested in?
We already include symbols from a bunch of builtin python modules https://github.com/benfred/py-spy/blob/25db5bc953d192a98cf485ba0cd01443f7048b1e/src/native_stack_trace.rs#L161-L173 . For instance profling this https://github.com/benfred/py-spy/blob/25db5bc953d192a98cf485ba0cd01443f7048b1e/tests/scripts/longsleep.py#L1-L9 with the native flags shows the time.sleep builtin to python:
Process 3333: python longsleep.py
Python v3.9.5 (/home/vagrant/miniconda3/bin/python3.9)
Thread 3333 (idle)
__select_nocancel (libc-2.17.so)
time_sleep (python3.9)
longsleep (longsleep.py:5)
<module> (longsleep.py:9)
vmprof doesn't handle callbacks afaik https://vmprof.readthedocs.io/en/latest/native.html#technical-design - after it detects a python frame it doesn't display any more native frames again.
Ah, it's nice to see the list here, thanks for the quick reply. Perhaps that whitelist could be made user-configurable?
I was specifically looking for PyOS_string_to_double (to figure out whether it's worth switching to one of the very fast native floating point parsers that exist nowadays). (So in my case, the extension is calling back into a C function implemented in libpython, not into an actual Python frame.)
Maybe we can just do what vmprof does? (stopping at PyEval_EvalFrame*)? That's a useful user-configurable flag IMO.
FWIW I ended up using a patched version of py-spy with just whatever entries I needed in WHITELISTED_PREFIXES (thanks for pointing it out, it's been very helpful), but being able to write e.g. py-spy record --whitelist-prefixes=foo,bar,baz,... -- the-python-process-invocation (with the default --whitelist-prefixes being what it is right now for compat) would be nice. (I am not wedded that the specific API either.)
In #497 I added a flag to disable most filtering. It's a bit noisy. For example, I see _start, Py_BytesMain, py_main_run_python, PyRun_AnyFileExFlags, and a few more above the main Python function. Also function_code_fastcall between Python calls. All these could be filtered out. But it does the job for me.
