lancedb
feat: allow general compression for string and binary types
This is an attempt to allow users to apply general compression such as zstd to compress string and binary types when they explicitly specify the relevant field metadata
This PR tries to address https://github.com/lancedb/lance/issues/2876
Right after submitting the PR, I realized that the relevant code in encoder.rs was significantly updated yesterday, making the PR largely irrelevant 🤣. I'll need some time to review the new code and figure out how the PR should be revised.
Right after submitting the PR, I realized that the relevant code in encoder.rs was significantly updated yesterday, making the PR largely irrelevant 🤣. I'll need some time to review the new code and figure out how the PR should be revised.
Sorry about that. Yes, some things have been shaking up a bit recently and I do think I disconnected zstd as part of that. My primary goal is to move scheduling out of the lower level "compressive encoders". I do want to reconnect zstd (and friends) as part of 2.1. I'll also look at this a bit today.
@niyue I've made an attempt in https://github.com/lancedb/lance/pull/2878