Mobius icon indicating copy to clipboard operation
Mobius copied to clipboard

Submitting a 32-bit application to Spark

Open LesiaCM opened this issue 7 years ago • 0 comments

I've asked this question on stackoverflow (https://stackoverflow.com/questions/48323609/submitting-a-32-bit-application-to-spark-via-mobius):

I would like to submit to Spark a 32-bit driver program. The driver program is the .NET Framework console app:

the app is rather simple and features a very simple testing code:

public static void Main(string[] args)
{

        var conf = new SparkConf(); 
        var sparkContext = new SparkContext(conf);

        var rdd = sparkContext.Parallelize(new List<string>() { "a", "b", "c", "d" });

        var response = rdd.Map(s => s).Collect();

}

After I properly submit this app to sparkclr (that is, to mobius which then passes it to Spark itself), I get the following exception:

System.Exception: JVM method execution failed: Static method collectAndServe failed for class 
org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, 
Type=JvmObjectReference, Value=11], )
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object 
classNameOrJvmObjectReference, String methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String 
methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Proxy.Ipc.RDDIpcProxy.CollectAndServe()
   at Microsoft.Spark.CSharp.Core.RDD`1.Collect()

The exception goes away when I build the app in x64. Also, the exception is not there when the app is built in x86 but does nothing (i.e. has no

  var rdd = sparkContext.Parallelize(new List<string>() { "a", "b", "c", "d" });
  var response = rdd.Map(s => s).Collect();

lines).

Is there any workaround to submit 32bit app to Spark? (In the original question I thought the issue could be with Java, but later found this was not the case).

With kind regards, Lesia

LesiaCM avatar Feb 13 '18 14:02 LesiaCM