linkedin
[all] Changed Kafka from LI's 2.4.1.65 to Apache's 2.4.1
Several code changes to make it work.
Introduced a new PubSubSecurityProtocol to replace all usage of Kafka's SecurityProtocol enum, since that one has a different package name between the Apache and LinkedIn forks of Kafka.
AK uses: org.apache.kafka.common.security.auth.SecurityProtocol While LI uses: org.apache.kafka.common.protocol.SecurityProtocol
TODO: Decide what to do about the producer flush API not having support for passing a time out in Apache Kafka.
This is a step in the direction of solving #998.
How was this PR tested?
GHCI... (needs more testing though...)
Does this PR introduce any user-facing changes?
- [x] No. You can skip the rest of this section.
- [ ] Yes. Make sure to explain your proposed changes and call out the behavior change.
I think PubSubSecurityProtocol is missing from the commit
Duh 😅 ... I pushed it up now... let's see if it works 💥 🤞
Closing temporarily. Will put up a separate PR to refactor the VeniceWriter's close routine, so that it stops relying on the flush with timeout API. Then I'll rebase the current PR on top of that other change.
Re-opened this PR, and rebased it on top of #1014, to solve the usage of the flush with timeout API, which does not exist in Apache Kafka. This should be good to go, I think...
Hey @FelixGV , do you have time to rebase from main and get your PR merged? 🚀
Hey @FelixGV , do you have time to rebase from main and get your PR merged? 🚀
Sorry, I missed your comment. The reason this PR is on hold is that we need to make some changes to our proprietary wrappers (MPs) in order to be able to pull in this change safely... @sushantmane got started and found some issues, and I'm planning to continue that work, but I haven't had a chance yet. Hoping to get to it soon.
But in the meantime, it means this PR is blocked...
I will close this one temporarily to jump on something else, but I do plan to get back to it soon-ish.