GraphScope icon indicating copy to clipboard operation
GraphScope copied to clipboard

Reduce memory usage during graph construction

Open siyuan0322 opened this issue 2 years ago • 0 comments

The graph construction requires a reasonably large amount of memory compared to the file size, maybe there is some room for optimization. For example, do some profiling and optimization in following scenario:

  • [ ] Multiple labels case, like in ldbc, which will have many unnecessary offset arrays.
  • [ ] Large vertex amount case, which will result in large global vertex map.
  • [ ] The runtime peak memory usage is 1.5 times of the final graph size, due to parallel sorting, etc. See if this could be reduced or give option to reduce the parallelism.

Some other optimization opportunities may came up during profiling.

siyuan0322 avatar Oct 11 '22 08:10 siyuan0322