Dominik Moritz

Thank you for the pull request. I think this is a good start but I would like to discuss a few options with you. > We did not use apache-arrow...

Thanks for the notes. I think I would personally still prefer Arrow since it encodes pretty efficiently and is well supported. It will make it a lot easier to maintain...

> As far as I know, Arrow serialization does not compress Arrow does support compression: https://arrow.apache.org/docs/python/generated/pyarrow.compress.html (JS support: https://issues.apache.org/jira/browse/ARROW-8674?jql=project%20%3D%20ARROW%20AND%20text%20~%20lz4). > assuming the serialization is as efficient as ours I suspect...

> There is a strong impedance mismatch between them and the python ecosystem. I'm surprised to hear that since Wes started both Pandas and Arrow. > Additionally, pyarrow tables are...

I see. It sounds like there are benefits to the custom serialization that everyone could benefit from. I'd be happy to merge this pull request then if you could help...

That all sounds great. In fact, there is a big advantage of separating out reusable components. There are Vega runtimes for Python notebooks not only here in ipyvega but also...

Thank you for pushing updated to the pull request. Let me know when it's ready for a review. I'll make this as draft for now.

I think I prefer to test it all together.

FWIW, Streamlit switched to using Arrow and it greatly reduced their code complexity: https://twitter.com/streamlit/status/1418637045468000256

Btw, Streamlit switched from their custom serialization to Arrow and got pretty good results: https://github.com/streamlit/streamlit/issues/239#issuecomment-935300084. I'm still very curious about the performance differences. If our custom serialization here is faster,...