mars icon indicating copy to clipboard operation
mars copied to clipboard

Optimize preparation of huge graphs

Open wjsi opened this issue 6 years ago • 2 comments

Is your feature request related to a problem? Please describe. When running huge graphs (~80k nodes) in Mars, runners have to wait for a long time (~20min) before the graph is tiled into chunks and actually start running. Therefore optimization is needed.

Describe the solution you'd like All time-consuming steps and possible optimizations:

  • [x] graph tiling (~10min -> 1.5min by applying #107)
  • [ ] initial placement (~6min)
  • [x] creating and distributing operands (~2min) -> eliminated by upgrading to v0.7

These steps can be optimized one by one.

wjsi avatar Jan 06 '19 07:01 wjsi

Can we give a isolated example to produce this process? So that we can dive into to profile and optimize.

qinxuye avatar Jan 07 '19 03:01 qinxuye

Now after #107, the tiling time has been reduced to 1.5min.

wjsi avatar Jan 07 '19 07:01 wjsi