Push compression metrics to StatsD/Datadog/Prometheus

[ellimist]

The Compressor class is pushing ActiveSupport notifications with metrics regarding the messages getting compressed.

However these don't get pushed to StatsD/Datadog, since there's no subscriber for them.

This PR adds the missing subscribers.

[ellimist]

@dasch I think I found a bug in the compression. I fixed it in the second to last commit: https://github.com/zendesk/ruby-kafka/pull/868/commits/3ad0c0447e96f13e8ca55fc407bc34ce0596ed84

Looks like the compression was removed in this commit 4 years ago? 😕

The functional tests started to fail though. I'm wondering if this is a red herring, or if the compression really wasn't being applied until now.

[dasch]

@dasch I think I found a bug in the compression. I fixed it in the second to last commit: 3ad0c04

Oh, should've read this before reviewing 😆

Looks like the compression was removed in this commit 4 years ago? 😕

The functional tests started to fail though. I'm wondering if this is a red herring, or if the compression really wasn't being applied until now.

No, I think compression was not being applied, but the way we do it now is not valid. The problem probably is that we can't just return the compressed data – we need to wrap it in some kind of record. See how it's being done in the legacy "message set" method... I'm actually not sure if the same method is used in the more recent protocol versions. Kafka has an... interesting protocol.

[dasch]

You might need to read up on how records are encoded: https://kafka.apache.org/documentation/#recordbatch

[github-actions[bot]]

Pull request has been marked as stale due to a lack of activity.