arrow-julia
arrow-julia copied to clipboard
apache
Official Julia implementation of Apache Arrow
I was surprised by this when I was writing a unit test that a function returned the expected structure. For example: ```julia julia> t = let io = IOBuffer(); Arrow.write(io,...
Context: The Arrow.jl 2.2.1 release changed the behavior of `@scopedenums` used in the `FlatBuffer` submodule, to remedy a type piracy issue. https://github.com/apache/arrow-julia/pull/267 This change introduced a couple of issues: 1....
``` julia> tab2 = (x = [rand(2,2) for _ in 1:10], ); julia> io2 = IOBuffer() IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1) julia> Arrow.write(io2, tab2) IOBuffer(data=UInt8[...], readable=true,...
It would be good to have similar support for InlineStrings in Arrow as in CSV.jl, ie habe string columns automatically stored and read in as the shortest possible InlineString
As discussed on Slack: ``` julia> using Arrow, DataFrames julia> Arrow.write("test.arrow", (a = [1, 2], b = Any[3, 4.5])) "test.arrow" julia> DataFrame(Arrow.Table("test.arrow")) 2×2 DataFrame Row │ a b │ Int64...
Attempt to solve #293, implementing `Tables.partitions()` for `Arrow.Table` turns out trivial enough, though I'm not clear what to do about companion test cases, documentation etc. I'm lazy to have not...
We already have a utility defined (`OrderedChannel`) that we use when writing record batches to ensure batches get _written_ in the same order they are provided; it makes sense to...
https://github.com/apache/arrow-julia/blob/614fce0a5d7db8fee078be32690c5220848538e2/src/table.jl#L276-L293 I see from above that record batches will be parsed (esp. decompression could be rather intensive computation workload) in parallel if the Julia runtime has multithread enabled, which is...
Hello, I have a custom type defined this way: ```julia struct Char8
I'm figuring out ways to have table columns processed by GPU, those coming from "arrow file" format files `mmap`ed for zero-copy. The full series can not fit into GRAM, while...