KoP 2.9.0 Plan

[BewareMyPower]

KoP 2.8.0 was released on Jun. 18th, it's a milestone that announced KoP is GA (generally available) for production. However, there're still some issues to solve or improve. This issue will list some important tasks.

Main tasks

• [x] Support for Kafka client 0.10 and 0.11 As far as I know, 0.10 and 0.11 clients are still widely used. There's a related issue: #539. BTW, though 0.9 client is also widely used, it may be too hard to be compatible with it. For example, 0.10 introduced the timestamp field to message format.
  • #605
  • #660
• [x] GC improvement KoP's performance is closed to the original Pulsar broker now, but there's still much room to improve. Like the GC problem mentioned in #538. It's because we don't make use of Netty Recycler for all objects in context currently.
  • #673

  NOTE: the left work's priority is low and currently mark it as done.

• [ ] Transaction GA (#39) There is an initial simple implementation of transaction in KoP 2.8.0 but it's not complete now. For example, client cannot set the timeout.
• [x] Support for authorization (#236) KoP only supports authentication currently. Due to no support for authorization (or just mixed into the authentication implementation), there're some related issues like #571 and #415.
  • #657
  • #662
  • #672
  • #671 Here're the optional tasks that may be not easy to implement before 2.9.0:
• [x] Support Kafka entry format for Pulsar client. It needs the change of Pulsar broker side and a PIP to provide the ability for Pulsar broker or client to parse other message format.
  • #632
• [ ] Kafka Streams unit tests migration. We have already migrated some unit tests from Kafka but there're still many tests that are not migrated. In addition, the Kafka Streams tests are sometimes not stable.

Important issues

Besides the tasks above, there're some other important issues that have high priority.

• [ ] Check log start offset (#580)
• [x] Support multiple listeners of one protocol (#574)
  • #646
  • #670
• [ ] Support a record with multiple batches (#537)

[junquero]

What about supporting Kafka clients > 2.0.0. I have tested with Apache Kafka 2.3.1 and it works, but it doesn't work with 2.4.0 and beyond. I'm using latest spring boot with Spring for Apache Kafka and I have had to downgrade to an old version to make it work with KoP, but the best alternative would be not to have to downgrade it.

Therefore, supporting not only old Kafka versions, but also some recent versions (from 2.4.x to 2.6.x) could be a possibility?

[BewareMyPower]

@junquero There're basic e2e tests for Kafka 1.0~2.6, see https://github.com/streamnative/kop/blob/252074234e9044eb9ee166e2a870f12d9712c12d/tests/src/test/java/io/streamnative/pulsar/handlers/kop/KafkaIntegrationTest.java#L91-L100

We may change the test method for different Kafka client versions. Currently it uses docker images to test, I think we can use a shade plugin later.

Back to your problem, could you give detail reproduce steps? Recently I did the KoP performance tests using Kafka client 2.6.0 and it worked well. See https://github.com/BewareMyPower/openmessaging-benchmark/blob/bewaremypower/deploy-kop/driver-kafka/pom.xml

[BewareMyPower]

BTW, I'm not familiar with Spring Boot :( It's better to reproduce your problem just using a Kafka client.

[junquero]

I have been able to connect latest kafka client 2.8.0 with KoP 2.8.0.2. The issue came from the fact that I used the docker image streamnative/platform-localrun which was updated 1 year ago.

I'd like to comment that I took some time to me to realize that when running the KoP in a container, the listeners can't be configured with 127.0.0.1:9092, but with 0.0.0.0:9092 if you want to access the protocol handler from outside the container.

May I suggest to add this hint/clarification in the KoP documentation?

[BewareMyPower]

@junquero Sorry for the late response. It seems that you've just edited your comment instead of adding a new comment or opening a new issue so that I cannot receive the notification.

For your question, if you're running KoP in a container, please use both kafkaListeners and kafkaAdvertisedListeners, see https://github.com/streamnative/kop/blob/master/docs/configuration.md#listeners. You should configure the external address with kafkaAdvertisedListeners. Though KoP only supports a single advertised listener currently, see https://github.com/streamnative/kop/issues/574 for more details.

BTW, if you have other questions, it's better to open a new issue instead of commenting in this issue. But it's welcome to giving your demands in this issue.

[eolivelli]

@junquero we recently added tests for Kafka 2.8.0 and Kafka 3.0.0 clients.

[eolivelli]

can we add the SchemaRegistry ? https://github.com/streamnative/kop/issues/771