gemini icon indicating copy to clipboard operation
gemini copied to clipboard

Spark: use native epoll transport on saving to DB

Open bzz opened this issue 7 years ago • 0 comments

If run on Spark cluster i.e on standalone mode, in Executors logs one can see this:

8/06/28 13:28:16 WARN NettyUtil: Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead.
java.lang.UnsatisfiedLinkError: Could not find prefix added to io.netty.util.internal.NativeLibraryLoader to get com.datastax.shaded.netty.util.internal.NativeLibraryLoader. When shading, only adding a package prefix is supported
	at com.datastax.shaded.netty.util.internal.NativeLibraryLoader.calculatePackagePrefix(NativeLibraryLoader.java:108)
	at com.datastax.shaded.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:120)
	at com.datastax.shaded.netty.channel.epoll.Native.loadNativeLibrary(Native.java:207)
	at com.datastax.shaded.netty.channel.epoll.Native.<clinit>(Native.java:65)
	at com.datastax.shaded.netty.channel.epoll.Epoll.<clinit>(Epoll.java:33)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at com.datastax.driver.core.NettyUtil.<clinit>(NettyUtil.java:68)
	at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:99)
	at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:769)
	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1410)
	at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:399)
	at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:161)
	at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154)
	at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154)
	at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
	at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
	at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
	at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:79)
	at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
	at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:210)
	at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:197)
	at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:183)
	at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
	at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	Suppressed: java.lang.UnsatisfiedLinkError: Could not find prefix added to io.netty.util.internal.NativeLibraryLoader to get com.datastax.shaded.netty.util.internal.NativeLibraryLoader. When shading, only adding a package prefix is supported
		at com.datastax.shaded.netty.util.internal.NativeLibraryLoader.calculatePackagePrefix(NativeLibraryLoader.java:108)
		at com.datastax.shaded.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:120)
		at com.datastax.shaded.netty.channel.epoll.Native.loadNativeLibrary(Native.java:210)
		... 28 more
18/06/28 13:28:16 INFO Cluster: New Cassandra host scylladb.default.svc.cluster.local/10.3.0.13:9042 added

Most probably Gemini build needs to be updated.

bzz avatar Jun 28 '18 14:06 bzz