[WIP] Provide geospatial support for Iceberg Spark
This PR depends on https://github.com/apache/sedona/pull/1828
Did you read the Contributor Guide?
-
Yes, I have read the Contributor Rules and Contributor Development Guide
-
No, I haven't read it.
Is this PR related to a JIRA ticket?
-
Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format
[SEDONA-XXX] my subject. -
No:
- this is a documentation update. The PR name follows the format
[DOCS] my subject - this is a CI update. The PR name follows the format
[CI] my subject
- this is a documentation update. The PR name follows the format
What changes were proposed in this PR?
GEOMETRY and GEOGRAPHY types has been accepted as part of Iceberg V3 spec. We are actively working on implementing geospatial support for the apache/iceberg project. This repository demonstrates how we'll integrate Sedona with Iceberg Spark to work with geometry and geography data. The overall idea is:
- We'll define an SPI interface
GeospatialLibraryProviderin apache/iceberg to allow third party libraries to integrate their geospatial support into iceberg-spark. - Sedona implements this
GeospatialLibraryProviderinterface in sedona-spark-iceberg module. - Iceberg will work with Sedona when iceberg-spark-runtime, sedona-spark-shaded and sedona-spark-iceberg are all installed into the Spark environment.
This PR is still in a proof-of-concept stage. It relies on several unreleased dependencies to work:
- parquet-format: for the new GEOMETRY and GEOGRAPHY logical types
- parquet-java: for the implementation of the parquet-geo standard
- iceberg: proof-of-concept implementation of the geo spec
How was this patch tested?
Tested locally
Did this PR include necessary documentation updates?
- Yes, I am adding a new API. I am using the current SNAPSHOT version number in
vX.Y.Zformat. - Yes, I have updated the documentation.
- No, this PR does not affect any public API so no need to change the documentation.
great!
my simple and invasive patch,https://github.com/freamdx/iceberg/commit/929dfae730d41516c77adf6801da99a01e410810 , i hope it will help you.