visualize static call graph

[bhpayne]

To aid in diagnosis of issues, a static call graph of compute.py and controller.py and the other .py files would be useful.

https://en.wikipedia.org/wiki/Call_graph

[bhpayne]

https://pycallgraph.readthedocs.io/en/master/ is good, but I only care about the functions for which I own the .py file.

sudo pip install pycallgraph
pycallgraph graphviz -- controller.py

https://pycallgraph.readthedocs.org/en/master/

https://pypi.python.org/pypi/pycallgraph/1.0.1

http://pycallgraph.slowchop.com/en/master/ http://pycallgraph.slowchop.com/en/develop/index.html

http://www.thinktoomuch.net/2007/06/06/python-call-graphs/

[bhpayne]

Problem: running the web app means the controller.py doesn't terminate, so the pycallgraph doesn't process the execution trace

[bhpayne]

Instead of running the web app and using pycallgraph for that, perhaps use

>>> import controller
>>> controller.index()

https://flask.palletsprojects.com/en/1.1.x/shell/

[bhpayne]

From @msgoff

an example of middleware in flask: https://stackoverflow.com/questions/42864788/apply-flask-logging-middleware-to-only-one-view

An example of pycall graph as django middleware -- https://vinta.ws/code/tools-for-profiling-your-python-django-project.html

[bhpayne]

@msgoff suggests using an AST walker, e.g., https://www.mattlayman.com/blog/2018/decipher-python-ast/

[bhpayne]

Ran https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/introspection/static_call_graph.py and produced PNGs that are too noisy to be useful.

[bhpayne]

In a Docker container I mounted proofofconcept/v7_pickle_web_interface/flask as /scratch and then ran

appuser@d88280cd47d6:/scratch/introspection$ python dump-ast.py ../compute.py | grep "FuncDef" -A1

to get all the function definitions. Playing with the printed output doesn't scale, so I modified your script to return "tree". https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/introspection/dump_ast.py

I wanted to work with the "tree" variable, so I entered a Python shell and ran

>>> from mypy.errors import CompileError
>>> from mypy.options import Options
>>> from mypy import defaults
>>> from mypy.parse import parse
>>> import dump_ast
>>> pyversion = defaults.PYTHON3_VERSION
>>> tree = dump_ast.dump('../compute.py', pyversion)

From there I was able to find a few useful outputs:

>>> for this_import in tree.imports:
...     print(this_import)

>>> for this_func in tree.defs:
...     print('=============')
...     if str(this_func).startswith('FuncDef'): print(this_func)

The next step would be to get the CallExpr from each "this_func" but I don't know how to do that.

[bhpayne]

@msgoff suggests looking at https://github.com/pytransitions/transitions/blob/master/examples/Graph%20MIxin%20Demo%20Nested.ipynb

[bhpayne]

Currently the website https://derivationmap.net/ runs in a Docker container and was developed using Flask to create webpages. If you're familiar with the "model-view-controller" paradigm, I've attempted to implement that in https://github.com/allofphysicsgraph/proofofconcept/tree/gh-pages/v7_pickle_web_interface/flask Specifically, the primary file is https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/controller.py which makes many calls to https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/compute.py when generating webpages.

Both of these files are complicated to navigate. Generating a picture of which webpages depend on which functions would be very useful for my project. I think a solution developed for this issue would be generically applicable to any project that uses Flask and the model-view-controller paradigm.

As an example of what I mean, take a look at the "start_new_derivation" route on line https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/controller.py#L1700 That function calls compute.initialize_derivation() which is defined on line https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/compute.py#L5035 which makes a call to dat = clib.read_db(path_to_db) which is defined in https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/common_lib.py#L99

Having a picture of this static call graph would help both with debugging and with assessing the impact of changes to workflow (refactoring).

I've made previous attempts at this task; see https://github.com/allofphysicsgraph/proofofconcept/tree/gh-pages/v7_pickle_web_interface/flask/introspection The files that are named "function_dependence_*" are relevant.

[bhpayne]

Comments migrated from old Trello board:

• https://github.com/jrfonseca/gprof2dot
• http://furius.ca/snakefood/
• https://ejrh.wordpress.com/2011/12/23/call-graphs-in-python/ and https://ejrh.wordpress.com/2012/01/31/call-graphs-in-python-part-2/