spark-dgraph-connector icon indicating copy to clipboard operation
spark-dgraph-connector copied to clipboard

Make Spark know the partitioning of the read data

Open EnricoMi opened this issue 3 years ago • 0 comments

The connector partitions the graph to allow Spark to read it in parallel. But Spark does not know anything about the partitioning. Say the connector partitions the graph by predicate and uid range, Spark would not know that and repartition / shuffle the read data if it wanted to join on partition or uid. If Spark would know the exact partitioning scheme, it could avoid un-needed shuffle steps.

Check to what extend Spark allows data sources to tell it about its partitioning.

EnricoMi avatar Nov 24 '21 14:11 EnricoMi