spark-dgraph-connector icon indicating copy to clipboard operation
spark-dgraph-connector copied to clipboard

Auto-adjust chunk size when gRPC message size is too small

Open EnricoMi opened this issue 4 years ago • 0 comments

The response from Dgraph mighht exceed the gRPC maximum message size. Due to skewed data, some partitions might see larger results than others. When a gRPC exception occurs indicating the message is too large, then this partition reader could reduce the chunk size for this partition and retry. This way we add resiliency and avoid that a Spark job dies half way.

EnricoMi avatar Nov 18 '20 20:11 EnricoMi