spark-connector
spark-connector copied to clipboard
pyspark with scala 2.13 jar doesn't work
This error will be thrown if you use the 2.13 fat jar with pyspark:
E py4j.protocol.Py4JJavaError: An error occurred while calling o50.save.
E : java.lang.NoSuchMethodError: 'scala.collection.convert.AsScalaExtensions$ListHasAsScala scala.jdk.CollectionConverters$.ListHasAsScala(java.util.List)'
E at io.weaviate.spark.Weaviate.inferSchema(Weaviate.scala:23)
E at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:90)
E at org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:280)
E at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:296)
E at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
Facing the same error, and then did some looking around. Found this write up that is using weaviate-spark-connector-assembly-v0.1.2.jar. https://weaviate.io/blog/details-behind-the-sphere-dataset-in-weaviate I am unable to locate the above jar in maven... so not able to test it out.