Feather file with compression and larger than RAM

[Moelf]

Last time I checked, mmap breaks down for files with compression. This is understandable because the compressed buffers clearly can't be re-interpreted without inflation.

But the larger the file is more likely to be compressed, can we decompress only a single "row group" (and only the relevant columns, of course) on the fly yet? -- this is for the case when a user is doing per-row iteration

if user access the table by tbl[range, range], then clearly, we might need to read more than one row-group and chop off head/tails depending on where the overlap is

[quinnj]

Hmmmm......we'll have to see what we can do here. I've had the idea for a while as a Tables.jl-wide feature to support projection/filter push down for sources in a generic way. That would translate really well to Arrow and would allow us to more easily avoid decompressing when not necessary. There's probably more we can do in the short-term though to avoid materializing when not needed.

[Moelf]

this is the whole thing we do un UnROOT.jl for a physics-community only thing called TTree, their next-gen storage called RNTuple is basically Apache Feather: https://indico.cern.ch/event/1208767/contributions/5083082/attachments/2523220/4340111/PPP_uproot_RNTuple.pdf#page=13

while we will get there eventually, in UnROOT I have the whole machinery basically:

• getindex -> find the row group ->
  • if not in cache, decompress and put in cache
  • if in cache, directly try to locate the slot

this way, at most one row group worth of data ever lives in RAM, in fact that's the minimal amount you need in RAM, because you can only know row number start-end for an entire row group and you have to count inside it.

but yeah, this is a whole thing in UnROOT.jl and it's mission-critical because our data are like, O(100) GB compressed all the time

[JoaoAparicio]

Hmmmm......we'll have to see what we can do here. I've had the idea for a while as a Tables.jl-wide feature to support projection/filter push down for sources in a generic way. That would translate really well to Arrow and would allow us to more easily avoid decompressing when not necessary. There's probably more we can do in the short-term though to avoid materializing when not needed.

Ping for comments on #412. This isn't the most general filter push down, but it does allow us to avoid unnecessary decompression.