feedback icon indicating copy to clipboard operation
feedback copied to clipboard
verified publisher icon observablehq

Support Apache Arrow in SQL cells for database clients

Open domoritz opened this issue 4 years ago • 3 comments

Is your feature request related to a problem? Please describe. Database clients have to return arrays off objects since Apache arrow is not supported.

Describe the solution you'd like Support Apache Arrow tables.

Additional context https://observablehq.com/@cmudig/duckdb could return arrow but has to materialize inefficient arrays of objects.

domoritz avatar Feb 01 '22 02:02 domoritz

Is there a plan for supporting this soon? It would be great to make the SQL support more robust and efficient.

domoritz avatar Feb 18 '22 17:02 domoritz

We are actively looking into this.

wiltsecarpenter avatar Feb 20 '22 05:02 wiltsecarpenter

@wiltsecarpenter did you make progress on this. Annie sent me a suggestion for the DuckDB client in https://observablehq.com/compare/357f1a71f976f173...ef33ae1a093daf79 and it's not using Arrow directly yet. Calling .toArray().map((r) => Object.fromEntries(r)) is super inefficient and I would like to avoid it.

domoritz avatar Jun 21 '22 14:06 domoritz