Nicholas Gates
Nicholas Gates
There was a discussion we had around whether filter evaluation could be replaced (merged) with projection evaluation. Currently they're subtly different. Filter evaluation is (m1: Mask) -> m2: Mask, where...
Actually the aim was to make vortex-arrow its own crate, at which point ToArrow _shouldn't_ be a compute function, and it can just be a regular Rust trait.
Am I right in thinking for you use case you will always be passing an Arrow target DataType?
I didn't think DataFusion handled dictionary / other encodings that weren't == the declared schema? Or am I out of date here?
Ahhh, gotcha. 1. Yes 100% 2. Yes 100%
I think we'd end up using the Rust-specific libraries for binding to those languages, same as Py03. e.g. https://github.com/neon-bindings/neon
So both session and Registry now hold dashmaps. I wonder if that's wrong, and we should actually hold HashMap? Or perhaps the deserialization logic should take an immutable snapshot of...
It's already gated by an environment variable. The whole point is you can switch them on/off using an env var without recompiling.
Nooo I like this!
@robert3005 do you think we want to use this + InlineExecutor?