delta
delta copied to clipboard
[Spark] Set up Python Protobuf codegen for Delta Connect
Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)
Description
Added the very first protobuf messages for DeltaTable, Clone and Scan.
This is the first PR for Delta Connect, which adds support for Delta's DeltaTable interface to Spark Connect. This is needed to support Delta table operations outside of SQL directly on Spark Connect clients.
This PR sets up the Python code generation for the Protobufs of Delta Connect. For this I created a new Buf workspace and I added a few initial Protobuf messages to confirm that everything works. This is the ground work of the project, before we move on to setting up the server and client library.
What we are doing here is similar to the Spark Connect's protobuf development guide.
How was this patch tested?
Added the check-delta-connect-codegen-python.py to the automated testing, making sure the Python Protobuf Generated codes stay in sync with the proto messages.
Does this PR introduce any user-facing changes?
No.
Failed tests are in org.apache.spark.sql.delta.perf.OptimizeGeneratedColumnSuite, seems unrelated to this PR.