clickhouse-go icon indicating copy to clipboard operation
clickhouse-go copied to clipboard

Support Compression Methods

Open gingerwizard opened this issue 3 years ago • 1 comments

Currently, we support lz4 only at a block level for both HTTP and native. ClickHouse supports more than this, specifically:

For native:

  • [x] zstd - this is supported in ch-go. Exposing should be simple. https://github.com/ClickHouse/clickhouse-go/pull/656

For HTTP via Accept-Encoding:

  • [x] zstd https://github.com/ClickHouse/clickhouse-go/pull/656
  • [x] ~lz4 (this would be a HTTP transport level, probably not needed given we support at block level)~
  • [x] br https://github.com/ClickHouse/clickhouse-go/pull/671
  • [x] gz https://github.com/ClickHouse/clickhouse-go/pull/664
  • [ ] xz
  • [x] deflate https://github.com/ClickHouse/clickhouse-go/pull/667

Also related:

  • [x] allow all compression methods via DSN and compress - currently bool. Support true(LZ4) or specific e.g. gzip. https://github.com/ClickHouse/clickhouse-go/pull/668

gingerwizard avatar Jul 08 '22 15:07 gingerwizard

xz compression is the remaining task here. I would rather not write a specific lib to achieve this - I don't think its higher enough value. The main option appears to be https://github.com/ulikunitz/xz. This isn't really focused on streaming use cases though and doesn't support Reset or Flush - we'd need to create a new reader and writer each time adding GC pressure. Whilst this is not critical, its also not compatible with our compression pool approach. There is a rewrite however which might address this https://github.com/ulikunitz/xz/tree/rewrite. Will wait for rewrite and reassess when its released.

If xz is an urgent feature for your usecase please raise here and we'll reconsider.

gingerwizard avatar Jul 23 '22 11:07 gingerwizard