mars
mars copied to clipboard
Optimize preparation of huge graphs
Is your feature request related to a problem? Please describe. When running huge graphs (~80k nodes) in Mars, runners have to wait for a long time (~20min) before the graph is tiled into chunks and actually start running. Therefore optimization is needed.
Describe the solution you'd like All time-consuming steps and possible optimizations:
- [x] graph tiling (~10min -> 1.5min by applying #107)
- [ ] initial placement (~6min)
- [x] creating and distributing operands (~2min) -> eliminated by upgrading to v0.7
These steps can be optimized one by one.
Can we give a isolated example to produce this process? So that we can dive into to profile and optimize.
Now after #107, the tiling time has been reduced to 1.5min.