java.lang.ClassCastException: scala.None$ cannot be cast to java.lang.Double

[caesarjuly]

trafficstars

Cast Error when transforming leapframe. I'm using the imputer from mleap-spark-extension and some fields of my data are null. I init these structs as ScalarType.Double.setNullable(true). But when I use the model to transform my generated leapframe, there comes the error.

java.lang.ClassCastException: scala.None$ cannot be cast to java.lang.Double
	at ml.combust.mleap.runtime.transformer.feature.Imputer$$anonfun$1.apply(Imputer.scala:14)
	at ml.combust.mleap.runtime.frame.Row$class.udfValue(Row.scala:241)
	at ml.combust.mleap.runtime.frame.ArrayRow.udfValue(ArrayRow.scala:17)
	at ml.combust.mleap.runtime.frame.Row$class.withValue(Row.scala:221)
	at ml.combust.mleap.runtime.frame.ArrayRow.withValue(ArrayRow.scala:17)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame$$anonfun$withColumn$1$$anonfun$apply$2$$anonfun$2.apply(DefaultLeapFrame.scala:54)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame$$anonfun$withColumn$1$$anonfun$apply$2$$anonfun$2.apply(DefaultLeapFrame.scala:54)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame$$anonfun$withColumn$1$$anonfun$apply$2.apply(DefaultLeapFrame.scala:54)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame$$anonfun$withColumn$1$$anonfun$apply$2.apply(DefaultLeapFrame.scala:53)
	at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
	at scala.util.Try$.apply(Try.scala:192)
	at scala.util.Success.map(Try.scala:237)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame$$anonfun$withColumn$1.apply(DefaultLeapFrame.scala:52)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame$$anonfun$withColumn$1.apply(DefaultLeapFrame.scala:49)
	at scala.util.Success.flatMap(Try.scala:231)
	at ml.combust.mleap.runtime.frame.DefaultLeapFrame.withColumn(DefaultLeapFrame.scala:48)
	at ml.combust.mleap.runtime.frame.SimpleTransformer$class.transform(Transformer.scala:112)
	at ml.combust.mleap.runtime.transformer.feature.Imputer.transform(Imputer.scala:11)
	at ml.combust.mleap.runtime.transformer.Pipeline$$anonfun$transform$2$$anonfun$apply$1.apply(Pipeline.scala:28)
	at ml.combust.mleap.runtime.transformer.Pipeline$$anonfun$transform$2$$anonfun$apply$1.apply(Pipeline.scala:28)
	at scala.util.Success.flatMap(Try.scala:231)
	at ml.combust.mleap.runtime.transformer.Pipeline$$anonfun$transform$2.apply(Pipeline.scala:28)
	at ml.combust.mleap.runtime.transformer.Pipeline$$anonfun$transform$2.apply(Pipeline.scala:28)
	at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
	at scala.collection.immutable.List.foldLeft(List.scala:84)
	at ml.combust.mleap.runtime.transformer.Pipeline.transform(Pipeline.scala:28)
	at com.zhihu.saturn.service.impl.RecommendationImpl.getSlotCardRank(RecommendationImpl.scala:31)
	at com.zhihu.saturn.thrift.RecommendationService$Processor$get_slot_card_rank.getResult(RecommendationService.java:181)
	at com.zhihu.saturn.thrift.RecommendationService$Processor$get_slot_card_rank.getResult(RecommendationService.java:165)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at com.zhihu.tzone.server.TProcessorInvocationHandler.invoke(TProcessorInvocationHandler.java:55)
	at com.sun.proxy.$Proxy19.process(Unknown Source)
	at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
	at com.zhihu.tzone.server.TZoneServlet.doPost(TZoneServlet.java:98)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

Some column values are null as showed below.

           null|                                   0.0|                               null|                               0.0|                              1.0|                              1.0|                            1.0|                            1.0|                       1.0|                                1.0|

Is the imputer cannot handle none value? I suppose imputer is designed to fill null values.

[hollinwilkins]

@caesarjuly Imputer is design with null values in mind. This is another problem with our docs you found. Should be using null instead of None for null values.

[hollinwilkins]

It could be that we didn't update imputer when we made that change. This is a good first project to investigate if you are looking for something to work on.

[caesarjuly]

@hollinwilkins Thanks for ur quick reply. I don't get it clearly. I use Option type to represent nullable values. Do u mean that I should directly use Null instead of None? It could be that we didn't update imputer when we made that change. Does this mean spark imputer already supported None but we don't?