spark-connector icon indicating copy to clipboard operation
spark-connector copied to clipboard

pyspark with scala 2.13 jar doesn't work

Open samos123 opened this issue 2 years ago • 1 comments

This error will be thrown if you use the 2.13 fat jar with pyspark:

E                   py4j.protocol.Py4JJavaError: An error occurred while calling o50.save.
E                   : java.lang.NoSuchMethodError: 'scala.collection.convert.AsScalaExtensions$ListHasAsScala scala.jdk.CollectionConverters$.ListHasAsScala(java.util.List)'
E                       at io.weaviate.spark.Weaviate.inferSchema(Weaviate.scala:23)
E                       at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:90)
E                       at org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:280)
E                       at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:296)
E                       at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)

samos123 avatar Feb 05 '23 01:02 samos123

Facing the same error, and then did some looking around. Found this write up that is using weaviate-spark-connector-assembly-v0.1.2.jar. https://weaviate.io/blog/details-behind-the-sphere-dataset-in-weaviate I am unable to locate the above jar in maven... so not able to test it out.

Kamalabot avatar May 01 '23 01:05 Kamalabot