spark-bigquery-connector icon indicating copy to clipboard operation
spark-bigquery-connector copied to clipboard

Use a helper to handle custom data type to avoid Spark ML dependency

Open irajhedayati opened this issue 2 years ago • 0 comments

In this PR, I added a helper class that does type lookup to see if we actually need to load SupportedCustomDataType. This class imports data types from Spark ML which forces the client code to import this library as well. More explanation of the issue can be found https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/599

In my application, I had:

  • Spark Core 3.1.3
  • Spark SQL 3.1.3
  • spark-bigquery-with-dependencies 0.24.2

I faced the issue and fixed it by adding Spark ML 3.1.3.

Then applied this fix and installed it locally as version 0.25.0. Updated spark-bigquery-with-dependencies to 0.25.0 and removed Spark ML 3.1.3. It then worked as expected.

Hence, I believe this patch is valuable to have in the code.

irajhedayati avatar Apr 22 '22 04:04 irajhedayati