vortex
vortex copied to clipboard
Async Expressions
This issue describes my current thinking around how to support async expressions in Vortex.
- Introduce an AsyncExprLayoutReader.
This will partition the expression into an async node, its sync children, and its sync parent expression. Async expressions are determined as their ID resolving to a LayoutEncoding in the LayoutRegistry, vs resolving to an ArrayEncoding in the ArrayRegistry.
- AsyncExprLayoutReader wraps its child layout in the layout loaded from the registry. The arrays returned from this layout are then wrapped in the parent sync expression.
We may wish to change the LayoutReader API to return streams, since some async expressions can blow up the in-memory size considerably and might want to return partial results.