incubator-graphar icon indicating copy to clipboard operation
incubator-graphar copied to clipboard

[Bug][Spark]: `GarDataSource` format does't support parallel write multiple dataframe

Open acezen opened this issue 7 months ago • 0 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Current Behavior

When try to write multiple data frame like:

    val offsetChunks = edgeDfAndOffsetDf._2.par
    offsetChunks.foreach { case (i, offsetChunk) =>
      FileSystem.writeDataFrame(
        offsetChunk,
        FileType.FileTypeToString(fileType),
        outputPrefix,
        Some(i),
        None
      )
      offsetChunk.unpersist()
    }

got error 截屏2024-01-19 15 24 29

Expected Behavior

support parallel write.

Minimal Reproducible Example

NA

Environment

  • Operating system:macOS M2
  • GraphAr version: v0.11.0

Link to GraphAr Logs

No response

Further Information

No response

acezen avatar Jan 19 '24 07:01 acezen