python API

[Liorba]

Hey, after seeing the europe spark summit lecture on magellan, you said python API will be available.

is there still a plan to add it? Thanks,

[halfabrane]

Hi @Liorba , I am in the process of cutting a 1.0.4 with Spark 1.5 The Python API will be available once that happens. Due to a class loading issue before Spark 1.5, the Python API was not functional unless you patched Spark... but with 1.0.4 that should be fixed. In fact, the Python API issue is what is keeping up the 1.0.4 release as some implementation changes in Spark 1.5 as far as Python goes has introduced an issue in Magellan's Python code which I m currently debugging. I'll let you know as soon as 1.0.4 is cut (which should be any day now)

[Liorba]

Thanks @halfabrane for your quick response, really looking forward to try the new version

[robbyki]

Hello,

Any update on when this will be ready?

Thank you

[GISDev01]

Hi @halfabrane, great news. I'm looking forward to trying this out as well!

[thenakulchawla]

Hi,

While importing the pyspark package in my server. Its killing the SparkContext. Error is given below. Please let me know how to resolve the given below issue

16/03/17 02:28:47 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144) at org.apache.spark.SparkContext.(SparkContext.scala:530) at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:59) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at py4j.Gateway.invoke(Gateway.java:214) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:745)