Request simple way to profile a single function without annotating code

[rswgnu]

I often have programs that do extensive setup and then call a particular function just once that dominates the runtime, so I would like to profile this single function and get cumulative execution times for each line in the that function. I would prefer to do this from the command-line without touching the Python code.

Is there any existing way to do this? If so, could you provide an example? If not, could you add it. I think it would be extremely useful.

I have tried using the code annotation method from the pprofile doc around code within a function that I know is executed during the run:

import pprofile
# Deterministic profiler
prof = pprofile.Profile()
with prof():
  result = my_func()
prof.print_stats()

but it does not produce a single line of output even though when I run pprofile from the commandline it provides voluminous output (and thus does not let me focus on the one function). This is on Windows 10 using Git bash as my shell. Thanks.

[rswgnu]

I have found a solution with a different package here. The profiling part of that article uses line_profiler which worked out of the box on Windows and let me quickly hone in on the performance of both a single function and specific subfunctions that it calls. It also runs pretty quickly.

You can either close this issue or leave it open so you pursue a solution. The above solution still requires annotating the code with what function to profile, so doing this dynamically is still an open issue.

[vpelletier]

For large projects, I would not recommend using textual format but cachegrind format, which you can then browse with qcachegrind (the kcachegrind version for windows). I see kcachegrind does not distribute binaries, which is a bit annoying. It is a very powerful tool (at least kcachegrind is, but I believe the only differences are KDE integration, like using the KDE standard "open file" dialog, maybe keyboard shortcuts which I don't use...), and I regularly browse profiling traces spanning over megabytes of source code with it, so it should be worth the hassle of building it from source.

If profiler overhead is an issue, then in addition to the above I recommend using pprofile's statistic profiling mode, which has very low overhead at the expense of trace details (no time stats, only hit stats accumulated by periodical sampling).

[rswgnu]

Thanks but I don’t think you addressed this issue request. The output pprofile produces already is fine for my needs. I just want a quick way without using any other tools to say ‘profile this single function’ from a python invocation. Would that be difficult to implement?

[vpelletier]

Something I forgot to mention in my previous reply is that I do not see why you did not get output, as I believe the code fragment you pasted is how I intend pprofile to be used in such way. Maybe something alters stdout in profiled program which would steal/hide print_stat's output ?

About adding an option to pprofile when used as a command, I think the difficult part is figuring out a syntax which would allow to unambiguously identify the callable to trigger profiling on. Of course, the easy cases are global functions and methods of global classes.

But if possible I would like to also support specifying local functions and method of local classes.

The next level would be being able to specify anonymous callables (lambdas) and non-native callables (I'm thinking of Zope's PythonScripts, for example: one would want to trigger when a specific script is being run, which happens when the easy-to-detect PythonScript..__call__ is called, but on the right instance. Zope is a rather artificial case, as I do not think running it under pprofile would be so helpful for PythonScript profiling).

Would you have any intuitive idea of how you would specify which callable to trigger profiling on ? Maybe I'm overthinking this, and it prevents me from finding a way.