incubator-graphar icon indicating copy to clipboard operation
incubator-graphar copied to clipboard

Published 20 hours ago • verified publisher icon apache

🛣️ Roadmap

Open acezen opened this issue 1 year ago • 0 comments

🛣️ Roadmap

Below is a high-level road map view for GraphAr to provide a sense of direction of where the project is going. This can change at any point and does not reflect many features and improvements that will also be included as part of the journey along this road map.

Format

  • Define format with protobuf (discuss and vote on [1][2])
  • Support multi-labels for vertex and edge
  • Standardizing the format v1 specification

C++ Libary

  • Format compatibility to v1
  • Make full use of feature of columnar format parquet/ORC to improve read/write performance
  • A simple out-of-core compute engine base on graphar

Java / Scala with Spark Library

  • Format compatibility to v1
  • Modularize the library: split to info/reader/writer...
  • Integrate with ldbc_snb_datagen_spark[3]

Python with PySpark

  • A new PySpark API that work with both Spark Classic and Spark Connect

Others

  • ETL CLI for graphar data [4]
  • More language binding
  • Construct a DataHub with GraphAr format

[1] https://lists.apache.org/thread/o5bqbhxvcbm6xqj1j1m2h7bhdnsvgsoy [2] https://lists.apache.org/thread/swg5qb35qxywt6w0k7oxt2srsvqnqgnh [3] https://github.com/apache/incubator-graphar/issues/463 [4] https://github.com/ldbc/ldbc_snb_datagen_spark

acezen avatar Dec 06 '22 02:12 acezen