Geojson import does not work on Spark 1.6

[jomach]

Hi Guys,

I was trying the new Library and it does not work on Spark 1.6

My Steps:

./spark-shell --jars /Users/jorge/Downloads/postgresql-9.4.1208.jar --driver-memory 4G --driver-cores 2 --packages harsha2010:magellan:1.0.3-s_2.10,com.databricks:spark-csv_2.11:1.4.0,com.databricks:spark-avro_2.10:2.0.1

import org.apache.spark.sql.{Row, SQLContext} val mageldf = sqlContext.read.format("magellan").option("type","geojson").load("/Users/jorge/Downloads/magellan/src/test/resources/geojson/polygon/example.geojson")

scala> mageldf.show 16/04/01 09:12:14 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericMutableRow cannot be cast to org.apache.spark.sql.Row .....

Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericMutableRow cannot be cast to org.apache.spark.sql.Row

[harsha2010]

@jomach magellan does not work with 1.6 yet. This is the main problem we are trying to solve right now: we can easily upgrade to 1.6 but cannot guarantee performance. So, it is requiring us some rearchitecture to work with Unsafe Rows and Encoders in order to guarantee performance when we do upgrade to Spark 1.6 I expect it will take us atleast a month to complete this upgrade (and in doing so we want to be ready for Spark 2.0 so likely you will get Magellan with Spark 2.0) In meanwhile, if Spark 1.5 works for you, I can cut a release with Spark 1.5 support this weekend

[jomach]

Hi, Thanks die the Reply. The most important is for example I already have a dataframe with a column containing a geojson polygon It would be nice to create a object from that so that I do not need to read always from disk.

Regards

Jorge Machado Www.jmachado.me

Am 01.04.2016 um 18:01 schrieb Ram Sriharsha [email protected]:

@jomach magellan does not work with 1.6 yet. This is the main problem we are trying to solve right now: we can easily upgrade to 1.6 but cannot guarantee performance. So, it is requiring us some rearchitecture to work with Unsafe Rows and Encoders in order to guarantee performance when we do upgrade to Spark 1.6 I expect it will take us atleast a month to complete this upgrade (and in doing so we want to be ready for Spark 2.0 so likely you will get Magellan with Spark 2.0) In meanwhile, if Spark 1.5 works for you, I can cut a release with Spark 1.5 support this weekend

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[qthurier]

Hi @harsha2010, I would be definitely interested in a release with Spark 1.5 support, this would avoid me few detours through postgis.. Many thanks, Quentin

[sabman]

@harsha2010 +1 to Spark 1.5 release Thanks!