spark icon indicating copy to clipboard operation
spark copied to clipboard

Published 20 hours ago • verified publisher icon dotnet

[BUG]: Spark 3.2 GetOrCreate failed for class '1'

Open ronnyek opened this issue 2 years ago • 3 comments

Describe the bug Following the mySparkApp tutorial but substituting in spark-3.2.0-bin-hadoop2.7, and the latest version of .net for spark nuget, I get errors trying to submit job

To Reproduce

Steps to reproduce the behavior:

  1. Follow directions on installing spark and .net for spark
  2. Follow directions to create mySparkApp (https://dotnet.microsoft.com/en-us/learn/data/spark-tutorial/create)
  3. Attempt to submit spark job with spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.1\microsoft-spark-3-2_2.12-2.1.1.jar dotnet bin\Debug\netcoreapp3.1\mySparkApp.dll
  4. Observe the following error
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/06/09 09:54:24 INFO DotnetRunner: Starting DotnetBackend with dotnet.
22/06/09 09:54:24 INFO DotnetBackend: The number of DotnetBackend threads is set to 10.
22/06/09 09:54:24 INFO DotnetRunner: Port number used by DotnetBackend is 52172
22/06/09 09:54:24 INFO DotnetRunner: Adding key=spark.jars and value=file:/C:/Users/WWS2/source/repos/mySparkApp/bin/Debug/netcoreapp3.1/microsoft-spark-3-2_2.12-2.1.1.jar to environment
22/06/09 09:54:24 INFO DotnetRunner: Adding key=spark.app.name and value=org.apache.spark.deploy.dotnet.DotnetRunner to environment
22/06/09 09:54:24 INFO DotnetRunner: Adding key=spark.submit.pyFiles and value= to environment
22/06/09 09:54:24 INFO DotnetRunner: Adding key=spark.submit.deployMode and value=client to environment
22/06/09 09:54:24 INFO DotnetRunner: Adding key=spark.master and value=local to environment
[2022-06-09T16:54:24.9525638Z] [LP5-WWS2-2DSA] [Info] [ConfigurationService] Using port 52172 for connection.
[2022-06-09T16:54:24.9719581Z] [LP5-WWS2-2DSA] [Info] [JvmBridge] JvMBridge port is 52172
[2022-06-09T16:54:24.9776739Z] [LP5-WWS2-2DSA] [Info] [JvmBridge] The number of JVM backend thread is set to 10. The max number of concurrent sockets in JvmBridge is set to 7.
22/06/09 09:54:25 INFO SparkContext: Running Spark version 3.2.0
22/06/09 09:54:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/09 09:54:25 INFO ResourceUtils: ==============================================================
22/06/09 09:54:25 INFO ResourceUtils: No custom resources configured for spark.driver.
22/06/09 09:54:25 INFO ResourceUtils: ==============================================================
22/06/09 09:54:25 INFO SparkContext: Submitted application: word_count_sample
22/06/09 09:54:25 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/06/09 09:54:25 INFO ResourceProfile: Limiting resource is cpu
22/06/09 09:54:25 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/06/09 09:54:25 INFO SecurityManager: Changing view acls to: WWS2
22/06/09 09:54:25 INFO SecurityManager: Changing modify acls to: WWS2
22/06/09 09:54:25 INFO SecurityManager: Changing view acls groups to:
22/06/09 09:54:25 INFO SecurityManager: Changing modify acls groups to:
22/06/09 09:54:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(WWS2); groups with view permissions: Set(); users  with modify permissions: Set(WWS2); groups with modify permissions: Set()
22/06/09 09:54:25 INFO Utils: Successfully started service 'sparkDriver' on port 52174.
22/06/09 09:54:25 INFO SparkEnv: Registering MapOutputTracker
22/06/09 09:54:25 INFO SparkEnv: Registering BlockManagerMaster
22/06/09 09:54:25 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/06/09 09:54:25 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/06/09 09:54:25 ERROR DotnetBackendHandler: Failed to execute 'getOrCreate' on 'org.apache.spark.sql.SparkSession$Builder' with args=()
[2022-06-09T16:54:25.6846003Z] [LP5-WWS2-2DSA] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
[2022-06-09T16:54:25.6846721Z] [LP5-WWS2-2DSA] [Error] [JvmBridge] java.lang.IllegalAccessError: Module 'unnamed module @56f123a6' no access to: package 'sun.nio.ch' which is not exported by module 'java.base' to module 'unnamed module @56f123a6'
        at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:338)
        at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
        at java.base/java.lang.invoke.MethodHandleResolver.getAdditionalBsmArg(MethodHandleResolver.java:465)
        at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:229)
        at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
        at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
        at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
        at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
        at org.apache.spark.SparkEnv$$$Lambda$520/0x000000001634ed68.apply(Unknown Source)
        at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$338/0x0000000015fa3db0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165)
        at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105)
        at org.apache.spark.api.dotnet.DotnetBackendHandler$$Lambda$326/0x0000000014807798.apply$mcV$sp(Unknown Source)
        at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:883)
Caused by: java.lang.IllegalAccessException: Module 'unnamed module @56f123a6' no access to: package 'sun.nio.ch' which is not exported by module 'java.base' to module 'unnamed module @56f123a6'
        at java.base/java.lang.invoke.MethodHandles$Lookup.throwIllegalAccessException(MethodHandles.java:876)
        at java.base/java.lang.invoke.MethodHandles$Lookup.checkClassModuleVisibility(MethodHandles.java:865)
        at java.base/java.lang.invoke.MethodHandles$Lookup.checkClassModuleVisibility(MethodHandles.java:782)
        at java.base/java.lang.invoke.MethodHandles$Lookup.checkClassAccess(MethodHandles.java:537)
        at java.base/java.lang.invoke.MethodHandles$Lookup.accessCheckArgRetTypes(MethodHandles.java:956)
        at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:311)
        ... 31 more

[2022-06-09T16:54:25.7307779Z] [LP5-WWS2-2DSA] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
 ---> Microsoft.Spark.JvmException: java.lang.IllegalAccessError: Module 'unnamed module @56f123a6' no access to: package 'sun.nio.ch' which is not exported by module 'java.base' to module 'unnamed module @56f123a6'
        at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:338)
        at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
        at java.base/java.lang.invoke.MethodHandleResolver.getAdditionalBsmArg(MethodHandleResolver.java:465)
        at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:229)
        at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
        at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
        at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
        at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
        at org.apache.spark.SparkEnv$$$Lambda$520/0x000000001634ed68.apply(Unknown Source)
        at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$338/0x0000000015fa3db0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165)
        at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105)
        at org.apache.spark.api.dotnet.DotnetBackendHandler$$Lambda$326/0x0000000014807798.apply$mcV$sp(Unknown Source)
        at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:883)
Caused by: java.lang.IllegalAccessException: Module 'unnamed module @56f123a6' no access to: package 'sun.nio.ch' which is not exported by module 'java.base' to module 'unnamed module @56f123a6'
        at java.base/java.lang.invoke.MethodHandles$Lookup.throwIllegalAccessException(MethodHandles.java:876)
        at java.base/java.lang.invoke.MethodHandles$Lookup.checkClassModuleVisibility(MethodHandles.java:865)
        at java.base/java.lang.invoke.MethodHandles$Lookup.checkClassModuleVisibility(MethodHandles.java:782)
        at java.base/java.lang.invoke.MethodHandles$Lookup.checkClassAccess(MethodHandles.java:537)
        at java.base/java.lang.invoke.MethodHandles$Lookup.accessCheckArgRetTypes(MethodHandles.java:956)
        at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:311)
        ... 31 more

   --- End of inner exception stack trace ---
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference jvmObject, String methodName, Object[] args)
   at Microsoft.Spark.Sql.Builder.GetOrCreate()
   at MySparkApp.Program.Main(String[] args) in C:\Users\WWS2\source\repos\mySparkApp\Program.cs:line 10
22/06/09 09:54:27 ERROR DotnetBackendHandler: Exception caught:
java.net.SocketException: Connection reset
        at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
        at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
        at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:883)
22/06/09 09:54:27 INFO DotnetRunner: Closing DotnetBackend
22/06/09 09:54:27 INFO DotnetBackend: Callback server has already been shutdown.
22/06/09 09:54:27 INFO ShutdownHookManager: Shutdown hook called
22/06/09 09:54:27 INFO ShutdownHookManager: Deleting directory C:\Users\WWS2\AppData\Local\Temp\spark-4a470003-cf7a-4f26-9f05-b50d2b0a0100

Expected behavior Honestly... I've never had this work local, and dont totally understand how its all fitting together, so the stack traces do very little for me. I can say I tried other versions and got this exact same error.

Additional context

  • verified that spark-submit --version reports version 3.2.0.
  • I downloaded winutils.exe for hadoop 2.7 and put that in the bin folder of the spark folder
  • I think I chose the right jar in the spark-submit command.
  • used netcoreap-3.1 for the app
  • verified I have Microsoft.Spark v2.1.1 installed which as I understand should support spark 3.2

ronnyek avatar Jun 09 '22 17:06 ronnyek

I have the same issue on spark-3.2.1-bin-hadoop2.7 Following this tutorial - https://learn.microsoft.com/en-gb/dotnet/spark/tutorials/get-started?tabs=windows

Exceptions:

  • JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments

  • JvmException: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x346d61be) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x346d61be

fatim avatar Nov 16 '22 14:11 fatim

I've downgraded OpenJDK from 19 to 11 on my machine and exceptions are gone.

fatim avatar Nov 21 '22 11:11 fatim

I want to note that I came across this as well on spark-3.2.1-bin-hadoop3.2 and @fatim's solution worked for me as well. I was on OpenJDK17 and downgraded to OpenJDK11 and it worked.

Rafnel avatar Aug 10 '23 19:08 Rafnel