azure-event-hubs-for-kafka
azure-event-hubs-for-kafka copied to clipboard
Azure
compression support for zstd
The documentation is for me quit unclear, it says "If compressed data is necessary, compressing your data before sending it to the brokers and decompressing after receiving it is a valid workaround. The message body is just a byte array to the service, so client-side compression/decompression will not cause any issues." Does this mean that i can use the kafka feature that does compression on client side (instead of passing "--config compression.type=gzip" at topic creation) and on producer side so the broker therefore does not need to compress by itself. Or does it mean kafka compression even in this approach will be blocked by the server. In that case this is a feature request. For WAN connections where IOT field gatways or on premise apps produce msgs to the iot hub , it's it mandatory to use compression (high volume) because of costs of reliable VPNs.
If I am understanding this request correctly, it is log compression that you are asking?
I mean compression of the messages going over the brokered tcp connections -not something on disk
yes today we do not compress the events. This needs to be in your clients. Let me take this as a user ask and we have more customers asking for the same, we will prioritize to implement it.
From my understanding kafka protocol clients already can do that on client side as it is also possible by config per topic on broker side while having producer->broker uncompressed. I am mainly asking for the support of (de)compression done on client side, but using the kafka protocol .
Producers(>=2.1): compression.type=zstd Consumers (<2.1): UNSUPPORTED_COMPRESSION_TYPE (error) Consumers(>=2.1): Working Brokers v 2.1 (Kafka).
Link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression
Is there still no plan to support this? Using open-telemetry collector for example and the kafka-exporter, the compression options do not work when using eventhubs for kafka. There's no way to compress the body data in this flow, so it would be great if EventHubs could support this!
Is this related to the options one gets in Azure Portal when, for example, setting up an input of type Event Hub in a Stream Analytics job and need to define the type of compression of events? Because there I also miss the option to use zstd. (Or at least give an option that would allow to read it raw and decompress it via user defined function)
