[question] Apache arrow support in altair ?

[djouallah]

I read that apache arrow is supported now in vega-lite using a separate vega-loader, is this something that altair will support.

for anyone who used Apache arrow, is the performance way better than csv, my current project has a dataset of 60000 line and it is a bit slow, specially for brushing

[jakevdp]

Arrow support in Vega/Vega-Lite is an open issue: https://github.com/vega/vega/issues/1300. Once it's available there, we will work to make it available in Altair

[djouallah]

Thanks, I was referring to this https://github.com/vega/vega-loader-arrow

[doug]

now that there is the vega-loader-arrow. What should support in altair look like?

[jakevdp]

It would first have to be supported by Vega-Lite.

[jakevdp]

Vega-Lite's schema does not yet allow the arrow format: https://github.com/vega/schema/blob/master/vega-lite/v4.11.0.json#L5526-L5541

[ivirshup]

It looks arrow data sources can be used by vega-lite now: https://observablehq.com/@vega/vega-lite-and-apache-arrow-no-plugin. If the data is served from a url and not passed as a javascript object, it looks like the plugin mentioned above will register the format (I think).

Is it a requirement for altair that this is implemented in the main vega/ vega-lite code bases?

[kimmolinna]

@jakevdp What is the situation with Apache Arrow and Arrow Table? @domoritz Am I correct that vega-lite supports Arrow nowadays?

[dhirschfeld]

I'm not sure what native arrow support altair has, but the vegafusion project integrates Apache Arrow DataFusion with altair

[kimmolinna]

I noticed that but I would like to use duckdb as a query engine. DuckdDB supports pandas but I prefer Apache Arrow.

[domoritz]

An Arrow JS Table object presents itself as an array of objects and Vega automatically supports it. This has been true for a while now. The arrow loader is only needed when you refer to an arrow file (in IPC format) by url.

[joelostblom]

@kimmolinna You can use duckdb via vegafusion as well https://vegafusion.io/duckdb.html. I'm closing this as it seems to me like the use cases requested here are covered by vegafusion, but please comment if I'm missing something.