delta icon indicating copy to clipboard operation
delta copied to clipboard

Published 20 hours ago • verified publisher icon delta-io

[Spark] Add Delta Connect Server Library

Open longvu-db opened this issue 1 year ago • 2 comments
trafficstars

Which Delta project/connector is this regarding?

  • [X] Spark
  • [ ] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [ ] Other (fill in here)

Description

This PR adds a skeleton for Delta Connect server library, and we add support for Scan to Delta's planner plugin.

How was this patch tested?

Added some basic tests for SparkConnectPlanner using the DeltaRelationPlugin and DeltaCommandPlugin plugins.

Does this PR introduce any user-facing changes?

No.

longvu-db avatar May 22 '24 17:05 longvu-db

@longvu-db do you have more details on what is Delta Connect?

felipepessoto avatar May 24 '24 22:05 felipepessoto

Hey @felipepessoto!

Delta Connect adds Spark Connect support to Scala and Python APIs of Delta Lake for Apache Spark. Spark Connect is a new project released in Apache Spark 4.0 that adds a decoupled client-server infrastructure which allows remote connectivity from Spark from everywhere. Delta Connect makes the DeltaTable interfaces compatible with the new Spark Connect protocol.

There are some issues regarding Delta Connect:

  1. https://github.com/delta-io/delta/issues/1570
  2. https://github.com/delta-io/delta/issues/1967

longvu-db avatar May 27 '24 19:05 longvu-db