Constructor failed for class org.apache.spark.api.csharp.CSharpR DD when called with 9 parameters

[csetzkorn]

I am using this code:

var sparkContext = new SparkContext(new SparkConf().SetAppName("ExcitingWarAndPeaceCount"));

var file = sparkContext.TextFile("file:///C:/war_and_peace.txt"); var warsCount = file.Filter(line => line.Contains("war")).Count(); var peaceCount = file.Filter(line => line.Contains("peace")).Count();

see also:

http://stackoverflow.com/questions/43449696/run-mobius-word-count-locally-in-debug-mode

and get this:

[2017-04-17T10:21:39.7031791Z] [PC76609LL] [Exception] [JvmBridge] JVM method ex ecution failed: Constructor failed for class org.apache.spark.api.csharp.CSharpR DD when called with 9 parameters ([Index=1, Type=JvmObjectReference, Value=11], [Index=2, Type=Byte[], Value=System.Byte[]], [Index=3, Type=JvmObjectReference, Value=8], [Index=4, Type=JvmObjectReference, Value=9], [Index=5, Type=Boolean, V alue=True], [Index=6, Type=String, Value=C:\VSProjects\SparkMobius\SparkMobius\b in\Debug], [Index=7, Type=String, Value=1.0], [Index=8, Type=JvmObjectReference, Value=10], [Index=9, Type=null, Value=null], ) at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStat ic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters )

[skaarthik]

What version of Spark and Mobius are you using?

[csetzkorn]

Thanks for reply.

Spark 2.1.0 built for Hadoop 2.7.3

spark-clr_2.11-2.0.200

[skaarthik]

https://github.com/Microsoft/Mobius/issues/611 Spark2.1 not supported yet

[csetzkorn]

Thanks! I am getting:

C:\spark-clr_2.11-2.0.200\runtime\scripts>sparkclr-shell [sparkclr-submit.cmd] Failed to find Spark jars directory. [sparkclr-submit.cmd] You need to build Spark before running this program.

for 1.6.1. I have downloaded the pre-build version. So the second message does not make sense. However, there is not explicit jars folder like for 2.1. Any help would be very much appreciated please.

Btw, when will this work for 2.1.*?

[skaarthik]

You do not have to downgrade from Spark 2.1 to 1.6.* to use Mobius. You can use Mobius with any Spark 2.0.* version. Mobius master branch is currently on this 2.0.2.

To use Mobius 1.6., you need any Spark 1.6. version. The error message indicates that SPARK_HOME variable is not set.

[csetzkorn]

Thanks. I downloaded:

spark-2.0.2-bin-hadoop2.7

Now I am getting:

{"JVM method execution failed: Constructor failed for class org.apache.spark.SparkContext when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=8], )"}

Any ideas? Also - let us say we have a cluster above version 2..0.* would the code I develop locally work there?

[csetzkorn]

Could anyone please be so kind and respond? I am using code above and after switching to 2.0.* I get above error message.

[skaarthik]

Mobius master branch is set to use Hadoop 2.6. Either try using Spark built with Hadoop 2.6 or try updating Mobius Hadoop references to 2.7.