Please write documentation!

[benwing]

Hello. I'm new to this project and just tried to test Scala pickling for possible use in a medium-sized NLP project (c. 32,000 lines of code). Scala pickling looks like an awesome project and I'd love to able to use it -- I have some relatively-simply-structured but very large and time-consuming-to-build objects for storing things like language models and ranking-model training data, and I want to serialize them once built to run different experiments on the same data.

But unfortunately there's almost zero documentation and so I have no idea how to do simple things like creating custom picklers. (I tried pickling a fairly simple class called SparseFeatureVectorFactory and I see it didn't pickle any of the subfields, so I need a custom pickler to specify them.) Is there any way that some documentation can be written up fairly quickly explaining basic usage and noting the main gotchas? The only docs I could find are Heather's OOPSLA paper, which is too technical to be useful as intro documentation. (Note in addition that even the "Basic Usage" section isn't quite right -- it doesn't say you need to write import binary._ or import json._ to select the pickling format.)

Thanks!

[nietaki]

Yeah, just a built scaladoc API would be a good start. The example from the project homepage brings more questions than it answers.

import scala.pickling._
import json._

val pckl = List(1, 2, 3, 4).pickle
//Unpickling is just as easy:

val lst = pckl.unpickle[List[Int]]

What's the type of pckl? What can it do? What types can you pickle?

I know I can read the automatic tests or add the library to my project and look inside (and I probably will), but having an API or a quickstart would save a lot of newcomers' time. Since you are asking in feedback in README, this is the very first thing that comes to mind.

[fuwaneko]

Not to mention it's not demonstrated how to unpickle from binary or json data read from file.

[raam86]

regarding binary, I have wasted quite a bit of time on this library and wanted to give up. However since like the rest of you I see the benefit in it I marched on.

The documentation is sparse because seemingly none is needed. You probably don't have to write custom anything.

The wording of the official site leave a lot to be desired so here's my take on things.

If pickling doesn't manage to automatically pickle your classes make sure you have the following:

import scala.pickling.Defaults._
import scala.pickling.binary._

Make sure you didn't import static etc'

these classes:

case class Event(id:UUID,`type`:String,timestamp:DateTime,hour_timebucket:DateTime,props:EventDetails)
case class EventDetails(properties:List[EventProp],interaction_id:String,session_id:String)

Were serialized with problem and with correct timestamps.

[fuwaneko]

@raam86 I have long since switched to Jackson/Protobuf. I actually dislike "idiomatic" Scala libraries nowadays, they usually have little to no documentation and it's totally a black magic inside which is hard to understand and modify.

[jsuereth]

@raam86 Please see a few of my branches incoming on fixes to the macros to help with the opaque nature that was there before.

ALSO, I was finally qualified to write this: https://github.com/scala/pickling/wiki/Writing-custom-Picklers-Unpicklers

That should help dramatically in writing custom picklers/unpicklers. I'll continue to provide docs and cleanups. If we can get the new macros into 0.11.x, you'll see a lot of cleanups incoming from that branch, as well as a lot more documentation on how to write picklers.

[raam86]

Thank you. Thats some really Great stuff

On Sunday, August 9, 2015, Josh Suereth [email protected] wrote:

@raam86 https://github.com/raam86 Please see a few of my branches incoming on fixes to the macros to help with the opaque nature that was there before.

ALSO, I was finally qualified to write this: https://github.com/scala/pickling/wiki/Writing-custom-Picklers-Unpicklers

That should help dramatically in writing custom picklers/unpicklers. I'll continue to provide docs and cleanups. If we can get the new macros into 0.11.x, you'll see a lot of cleanups incoming from that branch, as well as a lot more documentation on how to write picklers.

— Reply to this email directly or view it on GitHub https://github.com/scala/pickling/issues/98#issuecomment-129225532.