domoritz
Interested in any PRs?
👋 I've become convinced that there's a use for a library like arrow-wasm... not designed to be used directly from JavaScript (assuming no perf gain over JS) but instead to be used as a building block library for rust-wasm applications that want to use Arrow but don't want to manage wrapping core Arrow data structures to wasm themselves. And especially to simplify FFI between JS and Rust.
I started on a library at https://github.com/kylebarron/arrow-wasm using parts of your code as inspiration, but thought I should check if you were interested in any upstream PRs to this project 🙂
Absolutely although I haven't done much here as you can see so we could also shift development to your repo (I can archive this here and link to yours)! I think it would be even better if we could upstream code into apache/arrow. What do you think?
upstream code into apache/arrow
There's essentially two projects I'm working on:
-
arrow-js-fficode on the JS side of the boundary -
arrow-wasmon the Rust side of the Wasm boundary.
The former could potentially be upstreamed into Arrow JS, but I think it makes sense for now to be a separate library. For one, an application that mostly uses Wasm but might need minimal interop with JS could use that without bringing in the full Arrow JS dependency. And in a WIP refactor, it lets me handle all Arrow types, even the ones Arrow JS doesn't yet support, by having minimal objects holding the raw Arrow buffers.
The latter I don't think would make sense in Arrow JS. arrow-rs has built-in Python trait implementations, but not a Python API; I wouldn't guess they'd be interested in upstreaming an official wasm-bindgen binding. So I think that also makes sense as a separate library.
Building out a full JS API isn't yet a priority for arrow-wasm; first I'm focused on just handling table/record batch interop, where I can build geospatial table support on top in another library. So I'm not sure when my library will fully encompass your existing APIs in this repo