pickling
pickling copied to clipboard
scala
Schema evolution handling
Hi,
Scala Pickling looks highly promising so that I want to migrate an existing Eventsourced application which uses default Java serialization for event storage.
Currently the biggest concern other than #27 is schema versioning. The application requirement changes as time goes by, so eventually I'll need to add/rename/remove some fields from event messages, and then compatibility with previous versions of message will be an issue.
Is there any plan or recommendation about this?
+1 Maybe Option could be an option ? I hope there will be support for this.
Probably already checked, but FYI just in case: Schema evolution in Avro, Protocol Buffers and Thrift
At the moment it is necessary to create custom picklers to deal with schema evolution (for example, a custom pickler could unpickle older versions of a message, but create instances of newer versions).
However, it is possible to create new pickle formats which are more evolution-friendly, for example, by including something akin to Protobuf's field tags (the blog post linked above explains well how these tags help with schema evolution). Adding an evolution-friendly variant of the binary pickle format would be very useful, indeed.
Thanks for your advice, @phaller. At least, default parameter support will provide some basic compatibility out of the box, but it seems difficult without additional annotation until SI-5920 is resolved.
I decided to go with Protobuf for now but will keep eyes on Pickler looking for what I can contribute.
Since 0.10.0 came out I was looking at pickling for use to persist messages and was thinking about how schema evolutions would happen. Sadly this test I wrote didn't pass, but I'm mentioning it here to see if now that some of those PR's in the SI have been merged if it's likely for pickling to have support for handling default methods. It looks like it'll be fixed in scalac in 2.11.6?
https://github.com/adamdecaf/pickling-migration-test/blob/master/src/main/scala/Add.scala
Deleting fields works though!
In https://github.com/sbt/serialization we support Option fields that can be missing.
It would be totally awesome if schema evolution came about. It currently is the thing keeping us back from using pickling. @adamdecaf's outlined usecases pretty much sum up the hoped-for functionality for now.
protobuf/avro/thrift was mentioned before, but it is just not cutting it in a pure scala-based environment due to the lack of support for e.g. polymorphism, drastically cutting down the flexibility that scala offers.
So, very keen to get an update on this :smile:
