Py4JJavaError: An error occurred while calling o226.save. : java.lang.NoClassDefFoundError: scala/Product$class

[mednourconsulting]

i am getting this error Py4JJavaError: An error occurred while calling o226.save. : java.lang.NoClassDefFoundError: scala/Product$class in this step

write ratings data

ratings.write.format("es").save("ratings") num_ratings_es = es.count(index="ratings")['count'] num_ratings_df = ratings.count()

check write went ok

print("Dataframe count: {}".format(num_ratings_df)) print("ES index count: {}".format(num_ratings_es)) plz help me

[hung96ad]

I also got the same error

[pxike]

the student despair is real, i also had this issue, luckily for me i solved it on my own , the cause of the issue is that spark 3 is compiled in scala 2.12 , and the connectors in elasticsearch are compiled in 2.11, since elasticsearch website doesn't have that in their site (for some reason like wtf ?), u can search that in maven and install it manually, mvn dependency:get -DgroupId=org.elasticsearch -DartifactId=elasticsearch-spark-30_2.12 -Dversion=8.4.3 -Dtransitive=false then check the local maven repo /home/YourUserName/.m2/repository

either that or get a spark that is compiled in scala 2.11, which is spark 2.x versions but then the python version won't work cuz u will need python 3.4 or something which no longer supported , it's such a mean problem