ozhera icon indicating copy to clipboard operation
ozhera copied to clipboard

[enhancement] Kafka supports message compression

Open sadadw1 opened this issue 1 year ago • 0 comments

Currently, OzHera does not compress messages when sending them to Kafka. This can lead to excessive data consumption in certain cloud service providers that impose limits on data transfer, resulting in additional charges. We aim to reduce the consumption of data transfer by compressing messages. However, it is important to note that this compression may lead to increased CPU usage on both the producer and consumer ends.

Among the Kafka message compression algorithms available, gzip, snappy, and lz4 are considered effective. Based on a comparison from the official Facebook Zstandard website and other relevant sources,

image

we have chosen to use lz4 for message compression in Kafka.

sadadw1 avatar Dec 25 '23 10:12 sadadw1