arrow-julia icon indicating copy to clipboard operation
arrow-julia copied to clipboard

How to adapt `Arrow.Table` columns (naturally per record batch basis) into `CuArray`s for GPU processing?

Open complyue opened this issue 2 years ago • 0 comments

I'm figuring out ways to have table columns processed by GPU, those coming from "arrow file" format files mmaped for zero-copy.

The full series can not fit into GRAM, while each batch record can, so one thing is to get each contiguous chunk of the series data for upload to GPU memory in batching fashion. I've filed #293 to sort this out.

Another issue is arrow seems have intrinsic nullability, how can I adapt the array from arrow's data type to plain float64 series for uploading to GPU?

One more issue is with Julia's lacking of nanosecond precision timestamp by default, GPU can process those as int64, then how can I adapt arrow nanosecond precision ts to plain int64 arrays?

complyue avatar Mar 02 '22 15:03 complyue