incubator-graphar
incubator-graphar copied to clipboard
[Bug][Spark]: `GarDataSource` format does't support parallel write multiple dataframe
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
When try to write multiple data frame like:
val offsetChunks = edgeDfAndOffsetDf._2.par
offsetChunks.foreach { case (i, offsetChunk) =>
FileSystem.writeDataFrame(
offsetChunk,
FileType.FileTypeToString(fileType),
outputPrefix,
Some(i),
None
)
offsetChunk.unpersist()
}
got error
Expected Behavior
support parallel write.
Minimal Reproducible Example
NA
Environment
- Operating system:macOS M2
- GraphAr version: v0.11.0
Link to GraphAr Logs
No response
Further Information
No response