pickling
pickling copied to clipboard
Published
20 hours ago •
scala
scala
Running into problems when pickling Map and HashMap
steps (0.10.0)
scala-pickling> console
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51).
Type in expressions to have them evaluated.
Type :help for more information.
scala> import scala.collection.immutable
import scala.collection.immutable
scala> import scala.pickling._, Defaults._, json._
import scala.pickling._
import Defaults._
import json._
scala> immutable.HashMap(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.value
problem (0.10.0)
scala> println(immutable.HashMap(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.value)
{
"$type": "scala.collection.immutable.HashMap.HashTrieMap",
"bitmap": 2044127639,
"elems": {
"elems": [
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
},
{
"$type": "scala.collection.immutable.HashMap"
}
]
},
"size0": 20
}
original report
My data structure is stored as a Map and when I try to pickle it I run into some weird issues.
When pickling a immutable.HashMap using the following example code.
import scala.collection.immutable
import scala.pickling._
import json._
object Test {
def main(args : Array[String]) {
println(immutable.HashMap(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.toString)
}
}
as output I get
JSONPickle({
"tpe": "scala.collection.immutable.HashMap.HashTrieMap",
"bitmap": 2044127639,
"elems": {
},
"size0": 20
})
If I use instead the following snippet of code
import scala.collection.Map
import scala.pickling._
import json._
object Test {
def main(args : Array[String]) {
println(Map(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.toString)
}
}
I also end up with
JSONPickle({
"tpe": "scala.collection.immutable.HashMap.HashTrieMap",
"bitmap": 2044127639,
"elems": {
},
"size0": 20
})
only when I don't explicitly import Map do I end up with the correct output.
Most likely due to the recurring problem with anonymous classes / traits and their current unsatisfactory fix (https://github.com/scala/pickling/blob/2.10.x/core/src/main/scala/pickling/Compat.scala).
This is currently under investigation to find a more suitable solution.
Related to #32, #38 and possible others.
Yeah a fix would be very nice. because that makes it quite hard to work with nested data. Is there a workaround?
When i do not import Map at all I run into this compile time error:
scala:
while compiling: /home/wallnuss/Evo2DSim/src/main/scala/org/vastness/evo2dsim/evolution/SUSEvolution.scala
during phase: jvm
library version: version 2.10.3
compiler version: version 2.10.3
reconstructed args: -bootclasspath /opt/java/jre/lib/resources.jar:/opt/java/jre/lib/rt.jar:/opt/java/jre/lib/sunrsasign.jar:/opt/java/jre/lib/jsse.jar:/opt/java/jre/lib/jce.jar:/opt/java/jre/lib/charsets.jar:/opt/java/jre/lib/jfr.jar:/opt/java/jre/classes:/home/wallnuss/.m2/repository/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.jar -classpath /opt/java/jre/lib/jsse.jar:/opt/java/jre/lib/jfxrt.jar:/opt/java/jre/lib/jfr.jar:/opt/java/jre/lib/rt.jar:/opt/java/jre/lib/resources.jar:/opt/java/jre/lib/javaws.jar:/opt/java/jre/lib/plugin.jar:/opt/java/jre/lib/management-agent.jar:/opt/java/jre/lib/jce.jar:/opt/java/jre/lib/charsets.jar:/opt/java/jre/lib/deploy.jar:/opt/java/jre/lib/ext/sunjce_provider.jar:/opt/java/jre/lib/ext/sunec.jar:/opt/java/jre/lib/ext/dnsns.jar:/opt/java/jre/lib/ext/sunpkcs11.jar:/opt/java/jre/lib/ext/localedata.jar:/opt/java/jre/lib/ext/zipfs.jar:/home/wallnuss/Evo2DSim/target/classes:/home/wallnuss/.m2/repository/org/scala-lang/scala-reflect/2.10.0/scala-reflect-2.10.0.jar:/home/wallnuss/.m2/repository/org/jbox2d/jbox2d-library/2.2.1.1/jbox2d-library-2.2.1.1.jar:/home/wallnuss/.m2/repository/com/typesafe/akka/akka-actor_2.10/2.2.1/akka-actor_2.10-2.2.1.jar:/home/wallnuss/.m2/repository/com/typesafe/config/1.0.2/config-1.0.2.jar:/home/wallnuss/.m2/repository/com/intellij/forms_rt/7.0.3/forms_rt-7.0.3.jar:/home/wallnuss/.m2/repository/asm/asm-commons/3.0/asm-commons-3.0.jar:/home/wallnuss/.m2/repository/asm/asm-tree/3.0/asm-tree-3.0.jar:/home/wallnuss/.m2/repository/asm/asm/3.0/asm-3.0.jar:/home/wallnuss/.m2/repository/com/jgoodies/forms/1.1-preview/forms-1.1-preview.jar:/home/wallnuss/.m2/repository/jdom/jdom/1.0/jdom-1.0.jar:/home/wallnuss/.m2/repository/com/github/scopt/scopt_2.10/3.1.0/scopt_2.10-3.1.0.jar:/home/wallnuss/.m2/repository/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar:/home/wallnuss/.m2/repository/com/github/scala-incubator/io/scala-io-file_2.10/0.4.2/scala-io-file_2.10-0.4.2.jar:/home/wallnuss/.m2/repository/com/github/scala-incubator/io/scala-io-core_2.10/0.4.2/scala-io-core_2.10-0.4.2.jar:/home/wallnuss/.m2/repository/com/jsuereth/scala-arm_2.10/1.3/scala-arm_2.10-1.3.jar:/home/wallnuss/.m2/repository/org/scala-lang/scala-pickling_2.10/0.8.0-SNAPSHOT/scala-pickling_2.10-0.8.0-SNAPSHOT.jar
last tree to typer: Literal(Constant(anon$93))
symbol: null
symbol definition: null
tpe: Class(classOf[org.vastness.evo2dsim.evolution.Evolution$$anon$93])
symbol owners:
context owners: anonymous class anonfun$normalizeResults$2 -> package evolution
== Enclosing template or block ==
Template( // val <local $anonfun>: <notype>, tree.tpe=org.vastness.evo2dsim.evolution.anonfun$normalizeResults$2
"scala.runtime.AbstractFunction1", "scala.Serializable" // parents
ValDef(
private
"_"
<tpt>
<empty>
)
// 6 statements
DefDef( // final def apply(x$4: Tuple2): Tuple2
<method> final <triedcooking>
"apply"
[]
// 1 parameter list
ValDef( // x$4: Tuple2
<param> <synthetic> <triedcooking>
"x$4"
<tpt> // tree.tpe=Tuple2
<empty>
)
<tpt> // tree.tpe=Tuple2
Block( // tree.tpe=Tuple2
// 3 statements
ValDef( // case val x1: Tuple2
case <synthetic> <triedcooking>
"x1"
<tpt> // tree.tpe=Tuple2
Typed( // tree.tpe=Tuple2
"x$4" // x$4: Tuple2, tree.tpe=Tuple2
<tpt> // tree.tpe=Tuple2
)
)
LabelDef( // case def case6(): Tuple2, tree.tpe=Tuple2
()
If( // tree.tpe=Tuple2
Apply( // final def ne(x$1: Object): Boolean in class Object, tree.tpe=Boolean
"x1"."ne" // final def ne(x$1: Object): Boolean in class Object, tree.tpe=(x$1: Object)Boolean
null
)
Block( // tree.tpe=Tuple2
// 2 statements
ValDef( // val key: Int
<triedcooking>
"key"
<tpt> // tree.tpe=Int
Apply( // val _1$mcI$sp(): Int in class Tuple2, tree.tpe=Int
"x1"."_1$mcI$sp" // val _1$mcI$sp(): Int in class Tuple2, tree.tpe=()Int
Nil
)
)
ValDef( // val p2: Tuple2
<synthetic> <triedcooking>
"p2"
<tpt> // tree.tpe=Tuple2
Apply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=Tuple2
TypeApply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=()Tuple2
x1._2()."$asInstanceOf" // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=[T0 >: ? <: ?]()T0
<tpt> // tree.tpe=Tuple2
)
Nil
)
)
If( // tree.tpe=Tuple2
Apply( // final def ne(x$1: Object): Boolean in class Object, tree.tpe=Boolean
"p2"."ne" // final def ne(x$1: Object): Boolean in class Object, tree.tpe=(x$1: Object)Boolean
null
)
Block( // tree.tpe=Tuple2
ValDef( // val fitness: Double
<triedcooking>
"fitness"
<tpt> // tree.tpe=Double
Apply( // val _1$mcD$sp(): Double in class Tuple2, tree.tpe=Double
"p2"."_1$mcD$sp" // val _1$mcD$sp(): Double in class Tuple2, tree.tpe=()Double
Nil
)
)
Apply( // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=Tuple2
"matchEnd5" // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=(x: Tuple2)Tuple2
Apply( // final def ->$extension($this: Object,y: Object): Tuple2 in object Predef$ArrowAssoc, tree.tpe=Tuple2
Predef$ArrowAssoc.this."$minus$greater$extension" // final def ->$extension($this: Object,y: Object): Tuple2 in object Predef$ArrowAssoc, tree.tpe=($this: Object, y: Object)Tuple2
// 2 arguments
Apply( // implicit def any2ArrowAssoc(x: Object): Object in object Predef, tree.tpe=Object
scala.this."Predef"."any2ArrowAssoc" // implicit def any2ArrowAssoc(x: Object): Object in object Predef, tree.tpe=(x: Object)Object
Apply( // def box(x: Int): Integer in object Int, tree.tpe=Object
"scala"."Int"."box" // def box(x: Int): Integer in object Int, tree.tpe=(x: Int)Integer
"key" // val key: Int, tree.tpe=Int
)
)
Apply( // def box(x: Double): Double in object Double, tree.tpe=Object
"scala"."Double"."box" // def box(x: Double): Double in object Double, tree.tpe=(x: Double)Double
Apply( // final def f_norm$1(x: Double,abs_rel$1: Function1,total$1: Double): Double in class SUSEvolution, tree.tpe=Double
SUSEvolution$$anonfun$normalizeResults$2.this."$outer "."org$vastness$evo2dsim$evolution$SUSEvolution$$f_norm$1" // final def f_norm$1(x: Double,abs_rel$1: Function1,total$1: Double): Double in class SUSEvolution, tree.tpe=(x: Double, abs_rel$1: Function1, total$1: Double)Double
// 3 arguments
"fitness" // val fitness: Double, tree.tpe=Double
SUSEvolution$$anonfun$normalizeResults$2.this."abs_rel$1" // private[this] val abs_rel$1: Function1, tree.tpe=Function1
SUSEvolution$$anonfun$normalizeResults$2.this."total$1" // private[this] val total$1: Double, tree.tpe=Double
)
)
)
)
)
Apply( // case def case7(): Tuple2, tree.tpe=Tuple2
"case7" // case def case7(): Tuple2, tree.tpe=()Tuple2
Nil
)
)
)
Apply( // case def case7(): Tuple2, tree.tpe=Tuple2
"case7" // case def case7(): Tuple2, tree.tpe=()Tuple2
Nil
)
)
)
LabelDef( // case def case7(): Tuple2, tree.tpe=Tuple2
()
Apply( // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=Tuple2
"matchEnd5" // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=(x: Tuple2)Tuple2
Throw( // tree.tpe=Nothing
Apply( // def <init>(obj: Object): MatchError in class MatchError, tree.tpe=MatchError
new MatchError."<init>" // def <init>(obj: Object): MatchError in class MatchError, tree.tpe=(obj: Object)MatchError
"x1" // case val x1: Tuple2, tree.tpe=Tuple2
)
)
)
)
LabelDef( // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=Tuple2
"x" // x: Tuple2, tree.tpe=Tuple2
"x" // x: Tuple2, tree.tpe=Tuple2
)
)
)
ValDef( // private[this] val $outer: org.vastness.evo2dsim.evolution.SUSEvolution
private <local> <synthetic> <paramaccessor> <triedcooking>
"$outer "
<tpt> // tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
<empty>
)
DefDef( // final def apply(v1: Object): Object
<method> final <bridge>
"apply"
[]
// 1 parameter list
ValDef( // v1: Object
<param> <triedcooking>
"v1"
<tpt> // tree.tpe=Object
<empty>
)
<tpt> // tree.tpe=Object
Apply( // final def apply(x$4: Tuple2): Tuple2, tree.tpe=Tuple2
SUSEvolution$$anonfun$normalizeResults$2.this."apply" // final def apply(x$4: Tuple2): Tuple2, tree.tpe=(x$4: Tuple2)Tuple2
Apply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=Tuple2
TypeApply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=()Tuple2
"v1"."$asInstanceOf" // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=[T0 >: ? <: ?]()T0
<tpt> // tree.tpe=Tuple2
)
Nil
)
)
)
ValDef( // private[this] val abs_rel$1: Function1
private <local> <synthetic> <paramaccessor> <triedcooking>
"abs_rel$1"
<tpt> // tree.tpe=Function1
<empty>
)
ValDef( // private[this] val total$1: Double
private <local> <synthetic> <paramaccessor> <triedcooking>
"total$1"
<tpt> // tree.tpe=Double
<empty>
)
DefDef( // def <init>(arg$outer: org.vastness.evo2dsim.evolution.SUSEvolution,abs_rel$1: Function1,total$1: Double): org.vastness.evo2dsim.evolution.anonfun$normalizeResults$2
<method> <triedcooking>
"<init>"
[]
// 1 parameter list
ValDef( // $outer: org.vastness.evo2dsim.evolution.SUSEvolution
<param> <triedcooking>
"$outer"
<tpt> // tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
<empty>
)
ValDef( // abs_rel$1: Function1
<param> <synthetic> <triedcooking>
"abs_rel$1"
<tpt> // tree.tpe=Function1
<empty>
)
ValDef( // total$1: Double
<param> <synthetic> <triedcooking>
"total$1"
<tpt> // tree.tpe=Double
<empty>
)
<tpt> // tree.tpe=org.vastness.evo2dsim.evolution.anonfun$normalizeResults$2
Block( // tree.tpe=Unit
// 4 statements
If( // tree.tpe=Unit
Apply( // final def eq(x$1: Object): Boolean in class Object, tree.tpe=Boolean
"$outer"."eq" // final def eq(x$1: Object): Boolean in class Object, tree.tpe=(x$1: Object)Boolean
null
)
Throw( // tree.tpe=Nothing
Apply( // def <init>(): NullPointerException in class NullPointerException, tree.tpe=NullPointerException
new NullPointerException."<init>" // def <init>(): NullPointerException in class NullPointerException, tree.tpe=()NullPointerException
Nil
)
)
Assign( // tree.tpe=Unit
SUSEvolution$$anonfun$normalizeResults$2.this."$outer " // private[this] val $outer: org.vastness.evo2dsim.evolution.SUSEvolution, tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
"$outer" // $outer: org.vastness.evo2dsim.evolution.SUSEvolution, tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
)
)
Assign( // tree.tpe=Unit
SUSEvolution$$anonfun$normalizeResults$2.this."abs_rel$1" // private[this] val abs_rel$1: Function1, tree.tpe=Function1
"abs_rel$1" // abs_rel$1: Function1, tree.tpe=Function1
)
Assign( // tree.tpe=Unit
SUSEvolution$$anonfun$normalizeResults$2.this."total$1" // private[this] val total$1: Double, tree.tpe=Double
"total$1" // total$1: Double, tree.tpe=Double
)
Apply( // def <init>(): scala.runtime.AbstractFunction1 in class AbstractFunction1, tree.tpe=scala.runtime.AbstractFunction1
SUSEvolution$$anonfun$normalizeResults$2.super."<init>" // def <init>(): scala.runtime.AbstractFunction1 in class AbstractFunction1, tree.tpe=()scala.runtime.AbstractFunction1
Nil
)
()
)
)
)
== Expanded type of tree ==
ConstantType(value = Constant(anon$93))
uncaught exception during compilation: java.io.IOException
this does not work either:
import scala.pickling._
import json._
("test", Map(1->(2,3))).pickle
returns
JSONPickle({
"tpe": "scala.Tuple2[java.lang.String,scala.collection.immutable.Map[scala.Int,scala.Tuple2[scala.Int,scala.Int]]]",
"_1": "test",
"_2": {
"tpe": "scala.collection.immutable.Map[scala.Int,scala.Tuple2[scala.Int,scala.Int]]",
"elems": [
{
"tpe": "scala.Tuple2[scala.Int,scala.Tuple2[scala.Int,scala.Int]]",
"_1": 1,
"_2": {
"tpe": "scala.Tuple2$mcII$sp",
"_1": null,
"_2": null
}
}
]
}
})
the values in the tuple are both null instead of 2 & 3
Looks like this issue is due to the fact that specialized classes are not handled correctly. On Feb 8, 2015 7:07 AM, "eugene yokota" [email protected] wrote:
I was able to reproduce this problem using Pickling 0.10.0.
— Reply to this email directly or view it on GitHub https://github.com/scala/pickling/issues/63#issuecomment-73398665.
@phaller - Looking at this example @yigit provided, It is curious that the scala.Tuple2 at the top level seems to work fine but the Tuple2 as the value of the Map does not. Just thought I'd mention it in case that's a clue to fixing this.
Just some quick notes:
- I was able to correct the reported issue by providing a new hand-rolled ImmutableMap pickler which uses
TravPicklergenerator. - We still have the runtime issue, because the runtime pickler kicks in for
unpickle[Any]and this tries to look into class structure, which fails. - We need some additional runtime hooks to provide custom runtime picklers but NOT for concrete types, but rather type constructors (like Map[,]). We already have a mechanism in place specifically to handle Tuple2, but it's not extensible.
