Plans for implementing Pickle 5 serialization protocol?

[edoakes]

Hi all, I work on Ray and we have users who are interested in using Ion in conjunction with Ray. We use the pickle 5 serialization protocol internally, so if Ion were to support pickle 5 that would be ideal to provide zero-copy serialization for Ray users who want to use it :)

Any plans/thoughts here?

[tgregg]

Hi @edoakes, my first reaction is that supporting pickle 5 sounds like a good idea. I have very little existing knowledge about pickle, and I think the other maintainers of ion-python would likely say the same.

Would you be able to give us a rough idea of what would be required for ion-python to support pickle 5?

[edoakes]

That's great to hear!

Let me loop in our local serialization expert who will be able to give you a better idea :) cc @suquark

[suquark]

Here is the doc for pickle5: https://www.python.org/dev/peps/pep-0574/ I think to fully support pickle5, we need some C++/Cython code to expose the underlying buffer of python objects. I am not sure if it is a good idea, because it looks like Ion-Python is implemented with pure python code.

[tgregg]

It's implemented in pure python today, although we are aware that we are going to need to use a native code extension to achieve performance on par with other common python serialization libraries that use C extensions (like simplejson). We haven't been able to prioritize that work yet, though.

[suquark]

Yep, it is pretty good to have a native code extension. I also notice that ion-python uses bytearray, maybe it is useful for faster serialization. Let me dig a bit into it first.