polars icon indicating copy to clipboard operation
polars copied to clipboard

(de)serialize UDFs and dyn scans.

Open universalmind303 opened this issue 2 years ago • 1 comments

very much still a WIP, but an attempt at closing out #4569 , #6325, and #2979.

it's inspired by datafusion's Extension registry.

the general idea is that we expose 2 new functions

  • try_serialize
  • try_deserialize both of which take in a custom registry implementation. from there, it is up to the registry to figure out how to (de)serialize the trait objects, based on the names provided from the trait objects. The registry serializes them to a buffer, then we serialize that + the name as part of the SerializableLogicalPlan in place of Arc<dyn ...>.

I added a simple test, using a custom registry to de/serialize the LazyJsonLineReader that can give an idea on expected usage.

universalmind303 avatar Aug 14 '23 01:08 universalmind303

I've marked this as a draft PR for now, you can mark it as ready for review when you are ready.

orlp avatar Aug 14 '23 10:08 orlp

This PR hasn't been updated in a while and has gathered some conflicts. I will close it as stale for now, but feel free to open a new PR and continue the work!

stinodego avatar Feb 09 '24 16:02 stinodego