polars
polars copied to clipboard
(de)serialize UDFs and dyn scans.
very much still a WIP, but an attempt at closing out #4569 , #6325, and #2979.
it's inspired by datafusion's Extension registry.
the general idea is that we expose 2 new functions
try_serializetry_deserializeboth of which take in a custom registry implementation. from there, it is up to the registry to figure out how to (de)serialize the trait objects, based on the names provided from the trait objects. The registry serializes them to a buffer, then we serialize that + the name as part of theSerializableLogicalPlanin place ofArc<dyn ...>.
I added a simple test, using a custom registry to de/serialize the LazyJsonLineReader that can give an idea on expected usage.
I've marked this as a draft PR for now, you can mark it as ready for review when you are ready.
This PR hasn't been updated in a while and has gathered some conflicts. I will close it as stale for now, but feel free to open a new PR and continue the work!