pyroscope icon indicating copy to clipboard operation
pyroscope copied to clipboard

feat: add parquet compression option

Open liaol opened this issue 6 months ago • 0 comments

Parquet compression can save 50% disk or s3 usage.

CPU overhead (zstd with fatest level in production environment ):

  • write path(ingester flush, compression): < 1%
  • read path (querier?, decompression): 10%
  • compactor: decompression 15%,compression < 1%

Usage: add compression_algo and compression_level option to yaml or use -pyroscopedb.compression-level cli flag.

Test result for profile.Location (in pkg/phlaredb/compression_test.go), ZSTD Fastest is the best.

Algorithm       | Size(KB)     | Ratio(%)   | Compress(Write) Time | Decompress(Read) Time
----------------+--------------+------------+--------------+-------------
No Compression  | 1375.1       | 100.02     | 6.639104ms   | 20.057387ms
Snappy          | 341.7        | 24.86      | 8.657124ms   | 20.510525ms
LZ4 Fastest     | 340.7        | 24.78      | 6.876629ms   | 19.903987ms
LZ4 Default     | 301.1        | 21.90      | 104.934595ms | 13.72565ms
LZ4 Best        | 301.1        | 21.90      | 98.728766ms  | 12.769037ms
GZIP Fastest    | 239.0        | 17.38      | 6.377262ms   | 15.00412ms
GZIP Default    | 225.2        | 16.38      | 7.914216ms   | 14.851654ms
GZIP Best       | 244.2        | 17.76      | 2.103112012s | 14.874458ms
ZSTD Fastest    | 189.9        | 13.81      | 6.507433ms   | 16.559941ms
ZSTD Default    | 225.5        | 16.41      | 6.7972ms     | 13.503095ms
ZSTD Best       | 200.7        | 14.60      | 20.274187ms  | 13.017883ms

TODO:

  • Compress the symdb.

liaol avatar Jul 09 '25 06:07 liaol