Using Sparknlp with ray (https://docs.ray.io/en/latest/data/raydp.html) fails with classcastexception

[SrilekhaIG]

Trying to use sparknlp with Ray framework Tried to start the sparksession with Ray: spark = raydp.init_spark(app_name='Sparknlp', num_executors=2, executor_cores=2, executor_memory='4GB',configs={"spark.driver.memory":"16G", "spark.driver.maxResultSize": "0", "spark.kryoserializer.buffer.max": "2000M", "spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp-spark32_2.12:3.4.1"} )

However after running the sparknlp pipeline I have an issue with displaying the spark dataframe. I am following one of the basic examples: https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/NER_EN.ipynb

The exception I get at result.show() is :

Py4JJavaError: An error occurred while calling o294.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4) (10.59.192.207 executor 1): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.catalyst.expressions.objects.LambdaVariable.accessor of type scala.Function2 in instance of org.apache.spark.sql.catalyst.expressions.objects.LambdaVariable

Is there any configurations that I am missing? The error is for any of the complex datatypes in spark dataframe

[maziyarpanahi]

I have never used Spark NLP on Ray, not sure if Spark no Ray covers/supports 100% of all PySpark APIs.

@C-K-Loan @josejuanmartinez Have you guys ever tried Spark NLP on Ray framework? (or seen it done)

[SrilekhaIG]

I have opened an issue with ray team too. In case there are any updates will updare here

Here is the issue for reference: https://github.com/oap-project/raydp/issues/217

[SrilekhaIG]

Just an update on this, the spark.jars.packages config fails in raydp. The raydp team has given us a workaround with which am able to get the results. I have referenced this issue for raydp team in case they want to refer.

[josejuanmartinez]

Thank you @SrilekhaIG ! We appreciate this.

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 5 days

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 5 days