spark-dgraph-connector
spark-dgraph-connector copied to clipboard
Auto-adjust chunk size when gRPC message size is too small
The response from Dgraph mighht exceed the gRPC maximum message size. Due to skewed data, some partitions might see larger results than others. When a gRPC exception occurs indicating the message is too large, then this partition reader could reduce the chunk size for this partition and retry. This way we add resiliency and avoid that a Spark job dies half way.