arrow-julia
arrow-julia copied to clipboard
How to adapt `Arrow.Table` columns (naturally per record batch basis) into `CuArray`s for GPU processing?
I'm figuring out ways to have table columns processed by GPU, those coming from "arrow file" format files mmap
ed for zero-copy.
The full series can not fit into GRAM, while each batch record can, so one thing is to get each contiguous chunk of the series data for upload to GPU memory in batching fashion. I've filed #293 to sort this out.
Another issue is arrow seems have intrinsic nullability, how can I adapt the array from arrow's data type to plain float64
series for uploading to GPU?
One more issue is with Julia's lacking of nanosecond precision timestamp by default, GPU can process those as int64
, then how can I adapt arrow nanosecond precision ts to plain int64
arrays?