incubator-graphar icon indicating copy to clipboard operation
incubator-graphar copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Feat][Spark] Memory tuning for GraphAr Spark

Open acezen opened this issue 7 months ago • 0 comments

Is your feature request related to a problem? Please describe. Currently, GraphAr spark implementation can not handle very large dataset efficiency and got OOM error. We need to tuning the memory usage of GraphAr spark and make it scalable.

Describe the solution you'd like Tuning the memory usage with spark guide

  • [x] serialize RDD object with persist #326
  • [ ] tuning GC (especially when vertex chunk size is very small that would generate a large amount of rdd object)
  • [ ] tuning data structure

Additional context related to #119

acezen avatar Jan 16 '24 02:01 acezen