spark-bigquery-connector
spark-bigquery-connector copied to clipboard
Use a helper to handle custom data type to avoid Spark ML dependency
In this PR, I added a helper class that does type lookup to see if we actually need to load SupportedCustomDataType
. This class imports data types from Spark ML which forces the client code to import this library as well. More explanation of the issue can be found https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/599
In my application, I had:
- Spark Core 3.1.3
- Spark SQL 3.1.3
- spark-bigquery-with-dependencies 0.24.2
I faced the issue and fixed it by adding Spark ML 3.1.3.
Then applied this fix and installed it locally as version 0.25.0. Updated spark-bigquery-with-dependencies to 0.25.0 and removed Spark ML 3.1.3. It then worked as expected.
Hence, I believe this patch is valuable to have in the code.