kafka-streams-query icon indicating copy to clipboard operation
kafka-streams-query copied to clipboard

Published 20 hours ago • verified publisher icon lightbend

Update to Kafka Streams 2.0 API

Open jurgispods opened this issue 5 years ago • 14 comments

jurgispods avatar Dec 11 '18 08:12 jurgispods

Should we target upgrading to Kafka 2.2.0 ?

debasishg avatar Jun 28 '19 22:06 debasishg

At least one pull request committer is not linked to a user. See https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user#commits-are-not-linked-to-any-user

lightbend-cla-validator avatar Jul 03 '19 03:07 lightbend-cla-validator

@JustinPihony - I cannot find any blue mark beside the username as the linked to document specifies. Can u please help me find this problem ?

debasishg avatar Jul 03 '19 06:07 debasishg

@pederpansen Looks like your github profile doesn't have an email address. Hence the flagging by the validator. Could u please add an email address ?

debasishg avatar Jul 03 '19 06:07 debasishg

I have started integrating this PR and immediately facing issues with the latest versions ..

  • the examples depend on kafka-streams-scala which is no longer a valid artifact. We need to use the kafka streams scala APIs from Kafka itself, which are slightly diff from the ones in kafka-streams-scala
  • there have been changes in kafka APIs as well
  • ZkUtils is no longer a public class - hence we need to rewrite kafka-local-server using the new AdminClient

The effort doesn't look trivial. I am starting the work bt it will take some time to bring this to updated version of Kafka (2.2.0).

Thoughts ?

debasishg avatar Jul 03 '19 20:07 debasishg

@debasishg I've added an email address to the commits (if this is still needed).

jurgispods avatar Aug 02 '19 10:08 jurgispods

Thanks @pederpansen ..

debasishg avatar Aug 02 '19 15:08 debasishg

Hi @pederpansen,

Thank you for your contribution! We really value the time you've taken to put this together.

Before we proceed with reviewing this pull request, please sign the Lightbend Contributors License Agreement:

http://www.lightbend.com/contribute/cla

lightbend-cla-validator avatar Aug 12 '19 19:08 lightbend-cla-validator

It took me some time to get the CLA approved, which is now done.

@debasishg Shall we get on with the issue? Do you have a branch/PR for the work you started?

jurgispods avatar Sep 11 '19 12:09 jurgispods

@pederpansen I started work on this and then got pulled away elsewhere. Check out the branch upgrade-all where I did some work. The lib now builds ok with the latest versions of all dependencies. This should be good.

Started work on examples. Lots of stuff to work on here. I have changed some of the versions - will continue to work here.

One thing that I plan to do is to replace local-kafka-server with https://github.com/embeddedkafka/embedded-kafka ..

It will be great if u can help on any of these fronts ..

debasishg avatar Sep 11 '19 14:09 debasishg

I can also try to help on this task. Can you provide some advice on where I can help?

If I understand correctly we should work on the upgrade-all branch? We should convert examples and replace the local-kafka-server?

thanks

davideicardi avatar Aug 06 '20 18:08 davideicardi

@davideicardi You are correct. Unfortunately I have not been able to work on this because of serious lack of time. It would be great if u can take a look.

debasishg avatar Aug 07 '20 04:08 debasishg

I think that we should also use kafka-streams-scala library. There is any reason to not use it? It is probably a lot of work, but it should make sense...

davideicardi avatar Aug 09 '20 16:08 davideicardi

kafka-streams-scala has been merged to official kafka project (https://github.com/apache/kafka/tree/trunk/streams/streams-scala).

debasishg avatar Aug 10 '20 05:08 debasishg