clickhouse-go
clickhouse-go copied to clipboard
Support Compression Methods
Currently, we support lz4 only at a block level for both HTTP and native. ClickHouse supports more than this, specifically:
For native:
- [x] zstd - this is supported in ch-go. Exposing should be simple. https://github.com/ClickHouse/clickhouse-go/pull/656
For HTTP via Accept-Encoding:
- [x] zstd https://github.com/ClickHouse/clickhouse-go/pull/656
- [x] ~lz4 (this would be a HTTP transport level, probably not needed given we support at block level)~
- [x] br https://github.com/ClickHouse/clickhouse-go/pull/671
- [x] gz https://github.com/ClickHouse/clickhouse-go/pull/664
- [ ] xz
- [x] deflate https://github.com/ClickHouse/clickhouse-go/pull/667
Also related:
- [x] allow all compression methods via DSN and
compress- currently bool. Support true(LZ4) or specific e.g.gzip. https://github.com/ClickHouse/clickhouse-go/pull/668
xz compression is the remaining task here. I would rather not write a specific lib to achieve this - I don't think its higher enough value. The main option appears to be https://github.com/ulikunitz/xz. This isn't really focused on streaming use cases though and doesn't support Reset or Flush - we'd need to create a new reader and writer each time adding GC pressure. Whilst this is not critical, its also not compatible with our compression pool approach. There is a rewrite however which might address this https://github.com/ulikunitz/xz/tree/rewrite. Will wait for rewrite and reassess when its released.
If xz is an urgent feature for your usecase please raise here and we'll reconsider.