pyheatmagic
pyheatmagic copied to clipboard
csurfer
Accessing variables from previous notebook cells
Usually, one can access from the current notebook cell variables from previous cells (for instance, we define a in cell 2 and access it in 3). However, I can not get that to work when using heat (example in cell 4), which is sometimes annoying (especially when creating such variable can take up some time).
Full traceback (click to expand)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-4-6d5dc00d7528> in <module>()
----> 1 get_ipython().run_cell_magic('heat', '', 'print(a)')
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2085 magic_arg_s = self.var_expand(line, stack_depth)
2086 with self.builtin_trap:
-> 2087 result = fn(magic_arg_s, cell)
2088 return result
2089
<decorator-gen-124> in heat(self, line, cell)
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
186 # but it's overkill for just that one bit of state.
187 def magic_deco(arg):
--> 188 call = lambda f, *a, **k: f(*a, **k)
189
190 if callable(arg):
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/heat.py in heat(self, line, cell)
61
62 pyheat = PyHeat(tmp_file)
---> 63 pyheat.create_heatmap()
64 pyheat.show_heatmap(output_file=filename)
65 pyheat.close_heatmap()
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyheat/pyheat.py in create_heatmap(self)
44 """Method to define the heatmap using profile data."""
45 # Profile the file.
---> 46 self.__profile_file()
47 # Map profile stats to heatmap data.
48 self.__fetch_heatmap_data_from_profile()
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyheat/pyheat.py in __profile_file(self)
71 self.line_profiler = pprofile.Profile()
72 self.line_profiler.runfile(
---> 73 open(self.pyfile.path, 'r'), {}, self.pyfile.path)
74
75 def __get_line_profile_data(self):
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pprofile.py in runfile(self, fd, argv, fd_name, compile_flags, dont_inherit, globals)
606 try:
607 sys.argv[:] = argv
--> 608 return self.runctx(code, ctx_globals, None)
609 finally:
610 sys.argv[:] = original_sys_argv
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pprofile.py in runctx(self, cmd, globals, locals)
586 """Similar to profile.Profile.runctx ."""
587 with self():
--> 588 exec(cmd, globals, locals)
589 return self
590
/Users/Jean/Desktop/ipython_cell_input.py in <module>()
----> 1 print(a)
NameError: name 'a' is not defined
Would it be possible to add that feature? I have never worked on iPython extensions, but I would be glad to contribute to that one, if I'm pointed to the right part of code to improve.
Anyway, thank you for the great extension 👏
@nonatomiclabs: I will definitely look into the what would make the above possible keeping the API contract as simple as it is now. In the meantime, you can use https://github.com/csurfer/pyheat as a command line and use it on an entire file that way all your variables and the methods which access them are in one single file and you get an approximate idea of the bottlenecks in code.
P.S https://github.com/csurfer/pyheat is a pet project of mine which powers pyheatmagic.
Great, thank you very much.
About PyHeat, I'm not sure it's a real solution here, as:
- it's not as convenient as a simple iPython extension
- the reason I don't want to execute Heat/PyHeat on all of my code is that it can change the scale of the heat map. Let's say that I create a variable and it somehow takes 10 seconds to be created, but what is really interesting to me is to see timing of further code (which runs much quicker). Then, creating that variable will appear in dark red and all the other code will be in light yellow. If I could somehow exclude long operations from being included in the heat map, it wouldn't introduce such a bias in the heat map scale
Are either of the solutions in this stackoverflow thread (@needs_local_scope decorator & from IPython.core.magic import Magics) useful?
I've noticed issues with trying to refer to variables that are used outside of the function (for examples, variables defined in previous cells that are then passed into the function).
Is one way round this to allow those variables to be passed into the magic?
eg using a pattern along the lines of:
%%heat -v $myvar
def myfun(v):
pass
myfun(myvar)
where the -v switch lets you pass variables in to the block magic?
@csurfer Thanks for creating this extension. I am wondering if you ever found anything that might allow the variables in previous cells to be used?
You can also merge multiple cells just for doing "heat magic" selecting multiple cells and then shift+M, but you cant undo (well if you hit ctrl+shift+p you can search for undo, but it is not as good as others undoes in other programms.
The other is in git, save your work before doing the merge if you want to restore the cells as they where.
This issue is a definitive no-go for me (and I suspect many others), since any significantly complex analysis pipeline will start involving writing functions that you want to reuse from cells to cells (DRY code) and you don't want to start wrapping hundreds of lines within a single cell anytime you want to profile some snippet of code involving functions. That would just be a messy way to work. Seems to me that the problem is that the heat magic does not have access to local variables and functions. Maybe using the @needs_local_scope decorator may help? Example of usage: https://github.com/ipython/ipython/blob/23fb4ff6343e32b9d1921af4f9f1c58ffadd3376/IPython/core/magics/execution.py#L1063-L1084
