kafka_ex icon indicating copy to clipboard operation
kafka_ex copied to clipboard

Published 20 hours ago • verified publisher icon kafkaex

Support current kafka compression types

Open technomage opened this issue 5 years ago • 3 comments

Currently kafka supports more compression types by default than kafka_ex which can cause a client to fail when attempting to read such a topic. In particular confluent platform uses some of the newer compression types. At a minimum it would be nice to be able to configure kafka_ex to not die on such topics.

technomage avatar Aug 31 '19 18:08 technomage

@technomage which compression types are you using? The implementation of compression support so far has mostly been due to the needs of someone who has time and know-how to do the implementation :/

dantswain avatar Apr 26 '20 16:04 dantswain

Confluent uses one not currently supported which crashes a client trying to read from those topics.

technomage avatar Jul 08 '20 16:07 technomage

@technomage lz4 isn't yet supported, but a PR would be welcome:

https://cwiki.apache.org/confluence/display/KAFKA/Compression lz4 is type 3 - it would be added to https://github.com/kafkaex/kafka_ex/blob/76c752b82c48d04281b8fcf4ad4f14c4fbed335b/lib/kafka_ex/compression.ex to basically wrap the lz4 compress/decompress operations.

joshuawscott avatar Jul 15 '20 15:07 joshuawscott