apache
[SEDONA-133] Allow user-defined schemas in Adapter.toDf()
Did you read the Contributor Guide?
Yes, I have read Contributor Rules and Contributor Development Guide
Is this PR related to a JIRA ticket?
Yes, the URL of the assoicated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format [SEDONA-XXX] my subject.
What changes were proposed in this PR?
This expands the Adapter API to allow for users to convert to DataFrames with a given schema (for both SpatialRDD and JavaPairRDD).
User data is still stored in String format, so these new methods parse/cast the strings to whichever new data type is requested. This is similar to Spark's UnivocityParser, which is used to parse CSV files, but unfortunately that functionality is not exposed publicly so I created a barebones version here. I didn't cover all the data types, but tried to cover the key ones. This page details the encoders that Spark uses and may be helpful to understand the appropriate data types for conversion. We could expand to cover more data types later, or I'm open to it now if you request it.
This also adds some helper private methods and refactors a few common operations into their own functions.
How was this patch tested?
Added unit tests to confirm that SpatialRDD/JavaPairRDD -> DataFrame conversion works as expected.
Note: It doesn't seem to be common practice to test private methods, so I didn't add unit tests for the private methods I introduced. Their behavior is tested implicitly by the public functions.
Did this PR include necessary documentation updates?
Yes, I am adding a new API. I am using the current SNAPSHOT version number in since vX.Y.Z format.
Note: I don't see another appropriate place to change documentation. Please let me know if I missed this!
Questions
1.
Is the following behavior intentional? I don't have a strong geospatial background.
In the JavaPairRDD -> DataFrame test case (called "can convert JavaPairRDD to DataFrame with user-supplied schema"), you may notice that the left and right dataframes get switched. The SpatialJoinQuery has pointRDD on the left and polygonRDD on the right, but the final output has leftGeometry of type POLYGON, followed by the polygonRDD user data fields, and rightGeometry of type POINT, followed by the pointRDD user data (null).
@brianrice2
-
For your question
Is the following behavior intentional? I don't have a strong geospatial background., yes, it is intentional but it was due to my old silly design in GeoSpark... I didn't change it back due to backwards compatibility... Maybe we should change it back at some point... -
Please add some documentation in https://github.com/apache/incubator-sedona/blob/master/docs/tutorial/sql.md#convert-between-dataframe-and-spatialrdd to explain your API. Thank u!
- Thanks for providing that context! I'll create a followup Jira issue to discuss/prioritize separate from this issue. It would require extra care to be slotted into a major version update and communicated to users, so may be more trouble than it's worth. But at my work we do lots of join queries and primarily work with DataFrames, so I do come across this.
- Thanks for pointing that out—added some notes on converting with a schema