arrow-julia
arrow-julia copied to clipboard
apache
Official Julia implementation of Apache Arrow
Imagine a table `mytable` ~ 10x RAM size, the `Tables.partitions` is naturally defined as a lazy iterator: ```julia Tables.partitions(t::MyTable) = (t[range] for range in range_partition(t)) ``` But when subsequently calling:...
Hello, I'm currently dealing with a problem where I need to be able to append to a *.arrow file. The data needs to be added row by row because its...
`Arrow.write("testcsv.arrow", CSV.Chunks("data.csv"), ntasks=2)` with data.csv: ``` heading1,heading2,heading3 test1, test2, 3 test1, test2, 3 test1, test2, 3 test1, test2, 3 ... ``` throws this error: ``` Arrow.write("testcsv.arrow", CSV.Chunks("data/test/data.csv", ntasks=2)) ┌ Error:...
JuliaData/Arrow.jl has been transferred to apache/arrow-julia. See also: #265 and https://issues.apache.org/jira/browse/INFRA-22663 We discussed the development process for apache/arrow-julia on [email protected]. See the VOTE thread for summary of it: https://lists.apache.org/thread/6q333y875v7mfyl2g988b01hqtgr52pt apache/arrow-julia...
Hi! :) 👋 We observed a mysterious performance regression in some benchmarks when we started using `Arrow` in our codebase, and we have narrowed it down to type piracy on...
(the following is a slightly rewritten version of a slack post at the arrow-dev channel) I am interested in contributing to getting [flatc](https://google.github.io/flatbuffers/flatbuffers_guide_using_schema_compiler.html) support for julia. I have a [WIP...
Perhaps I'm misunderstanding `mmap`, but I would expect that `copy(df)` would be much faster the second time around, because `mmap` has by then lazily loaded the data into memory. On...
I want to serialize `Month` and `Year` etc. They do not have a unique representation in terms of seconds, so I try to serialize them directly. ```jl using Dates, Arrow...
```julia julia> using Arrow # v1.6.2 julia> table = (; col = [(; v=v"1"), (; v=v"2"), missing]) (col = Union{Missing, NamedTuple{(:v,), Tuple{VersionNumber}}}[(v = v"1.0.0",), (v = v"2.0.0",), missing],) julia> Arrow.Table(Arrow.tobuffer(table))...
Indexing an Arrow file that was written in multiple chunks with an out of bounds range sometimes produces a segfault. Tested on Arrow 2.2.0 and Julia 1.6.3. MWE: ``` using...
