Nicholas Gates

I'm surprised our current strategy doesn't create nested struct layouts? Is that what you're saying @a10y ? Ah, I think we lost that behavior at some point... Back when we...

I think variant types are separate from what we consider nested schemas (as variants are essentially untyped, there is no nesting of the schema itself). In that case, yes, we...

We should leave them in for now since they're very useful. But I do think they're under-specified in whether they're top-level columns, or include nested columns

I think your comment refers to https://github.com/vortex-data/vortex/issues/1835 @joseph-isaacs. That said, I don't think it's a wire-break since the zoned layout can later choose to support stats for struct dtypes (currently...

The default layout repartitions to the next multiple of 8K rows _above_ 1MB of uncompressed data. (You should be able to verify this with the 'cargo vx' command line) But...

Yep, so we've left open a slot for per-segment encryption and general purpose compression in the footer that we plan to support soon: https://github.com/spiraldb/vortex/blob/develop/vortex-flatbuffers/flatbuffers/vortex-file/footer.fbs But separately, we're digging into PCodec...

Absolutely for PCodec. I have a sort of background goal of reducing the surface area required to implement a new encoding, it's a little bit messy right now, but the...

Even with a tree of arrays, we need some sort of leaf node - in our case ByteBuffers. So while e.g. a PrimitiveArray could in theory store its data in...

I think I see where you're going... This is because PCodec is largely just a set of transformations that make the data more amenable to general purpose compression? For example,...

Refactor for arrays is done. You'll see the basic idea would be something like: ``` vtable!(Zstd) impl VTable for ZstdVTable { ... } ``` Hopefully there are sufficient docs to...