Yue

Hey @westonpace, I'm intrigued by the v2 format and I'm looking into adding support for general compression. I'd like to explore the possibility of encoding each page's buffer with zstd...

Thanks for the great suggestions. > we can hook it into the configuration mechanism later, once it's been developed Do we have a rough roadmap for when this might be...

Thanks for the insight. > RLE, [FSST](https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf), dictionary, and general compression are the next set Experimenting with general compression is useful in my scenario, especially since it can be applied...

@westonpace I have drafted a PR (https://github.com/lancedb/lance/pull/2368) to add support for compressing the value page buffer. Could you please review it to see if it fits well? And please let...

> Should we have a "prefer_simple_encodings" option which never reads into dictionary, even if the encoded form is dictionary This may be useful in some cases, for example, some frameworks...

I'm wondering if the current design proposal allows for reading partial and minimal dictionary tables to minimize I/O. In the example above, if only rows `[0, 2]` are requested, can...

@westonpace In the lance v2 format, we currently support plain encoding, dictionary encoding, and general compression. At present, plain encoding is the default option. General compression can be enabled via...

@westonpace > I have currently suggested that we use field metadata to make the decision Do you think such field metadata can be used to control whether other encodings (not...

Thanks so much for the explanation. > I think of these less as "encoding hints" and more as "extension types" Could you elaborate more on this part? Do you mean...

This feature is fantastic! Is there a way to flatten a `JSON` type column into multiple columns in ClickHouse? This would allow us to use modifiers such as `EXCEPT`, `COLUMNS`,...