Is it Work with spark 3?

[rubenssoto]

Hello,

Thank you for this great connector.

Is it work with spark 3? This connector is only supported by this repository? I found in apache bahir repository too.

[abhishekd0907]

This should work with Spark 3 because DataSourceV1 works with Spark 3. I haven't tried it myself though.

[DipeshV]

Hello @abhishekd0907 ,

I am running this connector on EMR 6.1.0 which has Spark 3.0. While running the stream I am getting the below error.

java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.streaming.sqs.SqsSourceProvider could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255) at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249) at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108) at scala.collection.TraversableLike.filter(TraversableLike.scala:347) at scala.collection.TraversableLike.filter$(TraversableLike.scala:347) at scala.collection.AbstractTraversable.filter(Traversable.scala:108) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:644) at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:194) ... 56 elided Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class at org.apache.spark.sql.streaming.sqs.SqsSourceProvider.(SqsSourceProvider.scala:26) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 73 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 80 `more```

Seems like the connector is not compatible with Spark 3.0 on EMR. Also, Spark 3.0 is based on scala 2.12 and the connector is build on scala 2.11.

Can you provide any highlights on this?

Thanks and regards, Dipesh Vora

[DipeshV]

Hey, I did some troubleshooting and found that the class org/apache/spark/internal/Logging$class is not the part of aws's version of spark-core and hence the error. I have downgarded to spark 2.4 and awaiting the fix on EMR Thanks, Dipesh

[abhishekd0907]

@DipeshV Thanks for the update. It will help in saving time for other folks

[FortBlox]

@DipeshV Do you know if there is an open ticket about this issue? I can't find anything regarding the internal logging class.