KAFKA-7632: Support Compression Level

[dongjinleekr]

Since I reworked KIP-390 from scratch, here I open a new PR.

Committer Checklist (excluded from commit message)

• [ ] Verify design and implementation
• [ ] Verify test coverage and CI build status
• [ ] Verify documentation (including upgrade notes)

[dongjinleekr]

@ijuma @guozhangwang Could you have a look when you are free? I refined the original work with a benchmark with a real-world dataset. As you can see in the updated KIP, this option can improve the producer's performance significantly. :pray:

[kkonstantine]

This PR implements KIP-390 but is still open and has several conflicts. Unless I hear from you by EOD Thursday, July 8 (pacific time), I'll remove it from the KIP list for 3.0

[dongjinleekr]

@kkonstantine Here it is - I rebased it onto the latest trunk. Could anyone review this PR? :pray:

[kkonstantine]

@tombentley do you think you'll be able to finish the review soon? If we had an approval I'd be willing to make an exception and consider this feature for inclusion to AK 3.0.

[ijuma]

@mimaison fyi, I think we need more votes in the KIP thread. Edit: Nevermind, this KIP (KIP-390) was adopted. I mixed it up with KIP-780.

[mimaison]

As far as I can tell this is implementing KIP-390 and it received 3 binding votes: https://lists.apache.org/thread/wnd6ky3kjzv9gkzc39qy5gg2pp20ovs1

[ijuma]

@mimaison yeah, I just edited my message. :)

[dongjinleekr]

Sorry for being late. Rebased onto the latest trunk, with resolving the issues raised by @mimaison @ijuma @junrao.

[clolov]

Hello @dongjinleekr! What is the current state of this pull request?

[ozturkberkay]

Hi, I was wondering if this PR would be prioritized in the foreseeable future given there has been a lot of work done already

[junrao]

One discussion of the PR is around choosing the default value for compression.level (https://github.com/apache/kafka/pull/10826#discussion_r917108950). This is tricky since it's specific to the compression type. I am wondering if it's better to adjust the KIP to support compression.{gzip|lz4|zstd}.level instead. This will make it more aligned with the future KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options#KIP780:Supportfinegrainedcompressionoptions-Zstd.1.

[mimaison]

Yes it would be good to get this over the line. @dongjinleekr Do you think you'll have time to update it? or would you need some help?

[mimaison]

I updated the KIP based on Jun's suggestion and sent an update to the VOTE thread.

[mimaison]

I opened https://github.com/apache/kafka/pull/15516 to implement this KIP.