Strategies for profiling functions in ipyparallel

[jakirkham]

I have a function that is going slow and I would like to figure out what is holding it up. I have profiled the underlying non-parallel function and it goes about as fast as it can on data of a size which it can handle. However, it really needs to work on blocks of data in parallel to get done in a reasonable amount of time and to work on data of any significance. So, I really need to profile it while it is running in parallel with ipyparallel. Are there any suggested ways to go about this? Is it possible to use some existing tool like line_profiler? If not, what other tools might be look at? Does ipyparallel have any tricks for doing this sort of thing?

[jakirkham]

I ended up trying something like this ( http://seiferteric.com/?p=277 ) with some minor tweaks. See related SO question ( http://stackoverflow.com/q/11743203 ).

import cProfile
import atexit
import os,sys
def ProfExit(p):
  p.disable()
  prof_f_name = "%d"%os.getpid()
  p.dump_stats("/root/profiles/%s"%prof_f_name)
profile_hook = cProfile.Profile()
atexit.register(ProfExit, profile_hook)
profile_hook.enable()