GraphScope icon indicating copy to clipboard operation
GraphScope copied to clipboard

[GIE-GAIA] Profiling for GAIA

Open BingqingLyu opened this issue 3 years ago • 3 comments
trafficstars

What do these changes do?

  1. Provide profile tools in GAIA-IR, to show information of time cost, number of intermediate results in each operator, etc.
  2. Provide graphviz to visualize the profiling results

Related issue number

Fixes #1805

BingqingLyu avatar Jul 12 '22 02:07 BingqingLyu

Codecov Report

Merging #1824 (55f04f0) into main (67a45aa) will decrease coverage by 31.15%. The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##             main    #1824       +/-   ##
===========================================
- Coverage   71.48%   40.32%   -31.16%     
===========================================
  Files          89       89               
  Lines        9847     9847               
===========================================
- Hits         7039     3971     -3068     
- Misses       2808     5876     +3068     
Impacted Files Coverage Δ
...ython/graphscope/tests/unittest/test_cython_ast.py 0.00% <0.00%> (-100.00%) :arrow_down:
...ython/graphscope/tests/unittest/test_serailaize.py 0.00% <0.00%> (-100.00%) :arrow_down:
python/graphscope/analytical/udf/patch.py 3.47% <0.00%> (-96.53%) :arrow_down:
python/graphscope/tests/unittest/test_lazy.py 0.00% <0.00%> (-96.30%) :arrow_down:
...thon/graphscope/tests/unittest/test_scalability.py 0.00% <0.00%> (-96.16%) :arrow_down:
python/graphscope/tests/unittest/test_app.py 0.00% <0.00%> (-95.66%) :arrow_down:
...hon/graphscope/tests/unittest/test_create_graph.py 0.00% <0.00%> (-92.00%) :arrow_down:
python/graphscope/tests/unittest/test_graph.py 0.00% <0.00%> (-85.57%) :arrow_down:
python/graphscope/tests/unittest/test_context.py 0.00% <0.00%> (-81.04%) :arrow_down:
python/graphscope/analytical/udf/wrapper.py 27.27% <0.00%> (-72.73%) :arrow_down:
... and 37 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 67a45aa...55f04f0. Read the comment docs.

codecov-commenter avatar Jul 12 '22 03:07 codecov-commenter

Consider use graphviz to visualize the results. Additionally, we can a bi case to demonstrate the result of how this profiling can help optimization.

Have added graphviz for visualizing the results.

BingqingLyu avatar Jul 18 '22 13:07 BingqingLyu

@bmmcq Hi, bingqing wrote some codes for profiling Pegasus, by logging the run time, size of output/communication data of each operator. See if you have any good advice.

longbinlai avatar Aug 12 '22 12:08 longbinlai