Spike on a new API for Kafka Streams support

[bobby]

My team ran into some issues when using KafkaStreams with Micronaut on a current project:

• Returning KStream from factory methods seemed odd and arbitrary (what if my topology only uses KTable or GlobalKTable ?), and the KStream bean itself is never used by the Factory
• Running multiple KafkaStreams jobs never quite worked right, since the ConfiguredStreamBuilder and KStream weren't named or specified in the KafkaStreams factory method

So we reworked the API a bit, and this new implementation has solved these issues. Now the API looks like:

    @Named("myStream")
    @Singleton
    fun myStream(@Named("my-stream") configuration: KafkaStreamsConfiguration) : ConfiguredStreamBuilder {
        val builder = ConfiguredStreamBuilder(configuration)
        ...
        return builder
    }

If this API looks good to the maintainers, I'll be happy to update the documentation, etc.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

:x: bobby
:x: tetriscode
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[graemerocher]

Test failures seem unrelated and resolved by another PR

[graemerocher]

The changes seem ok to me, only think is they are breaking changes. I wonder if there is way to maintain some form of backward's compatibility. Either way we need to alter the version in https://github.com/micronaut-projects/micronaut-kafka/blob/master/gradle.properties#L1 to 1.2.0.BUILD-SNAPSHOT

[bobby]

@graemerocher If we're backwards-breaking, wouldn't we bump to 2.0.0.BUILD-SNAPSHOT, as per SemVer?

[bobby]

I've rebased with my correct email address in the commit author, to please the CLA bot.

[ctoestreich]

@bobby Would there be any doc updates for your changes?

[bobby]

@ctoestreich Yes, there will need to be doc changes given the new API. This new API will also enable Kafka Streams testing, but I may add that plus testing docs in its own PR.

[graemerocher]

@bobby could you rebase the changes and provide documentation? We do use semantic versioning so I will need to split out a 1.1.x branch before merging this commit. Thanks.

[graemerocher]

@bobby any more thoughts on documentation? I am thinking that may we should just bump the kafka streams module to 2.0 or even split it into a separate repo. That way the core library can continue to evolve. Thoughts?

[bobby]

Yes, I think at least the kafka-streams module will need to bump to 2.0.0 given the changes, but can probably stay in same repo?

I’m working on the docs now.

Cheers, Bobby

Sent from my iPhone

On May 27, 2019, at 10:22 AM, Graeme Rocher [email protected] wrote:

@bobby any more thoughts on documentation? I am thinking that may we should just bump the kafka streams module to 2.0 or even split it into a separate repo. That way the core library can continue to evolve. Thoughts?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[bobby]

While testing an application with several instances of KafkaStreams, we've run up against some difficulties. I'm going to attempt to resolve these difficulties via a few more minor API tweaks in the coming days.

[ctoestreich]

@bobby I also ran into this a bit recently. I am available to help with your branch a bit if you want to resurrect this PR. I spent some time during my last PR to add health trying to dissect the KafkaStreamsFactory as I needed stuff there to relate a kstream to the configuration used to build it for metadata on the health endpiont.

[graemerocher]

@ctoestreich let me know if you plan to contribute this, since we plan to do a release soon. I have updated the module to Kafka 2.5.0 in the meantime

[ctoestreich]

Spending some time today revisiting @bobby code. @bobby @tetriscode what we’re the remaining open issues?

[rgonciarz]

Hi @bobby @graemerocher @ctoestreich and others, is there any chance to fix this PR and merge it? or to do a serious changes in Micronaut's Kafka Streams configuration/DSL? I'm trying to escape from Spring stack. Micronaut is very promising but the current support for Kafka Streams is far away from being perfect. What I like with Spring approach is that you may easily create several typologies within one project by exposing simple bean bi/functions/consumers using their Spring Cloud Kafka Streams binder (functional style). Except that they offer a configuration hell in yaml files and you loose control over the streams. Recently I've noticed a new library with very promising API/DSL on the market that is trying to simplify Kafka Streams configuration: https://www.azkarrastreams.io/docs/getting-started/ I didn't have a chance to try it, but it seems to introduce a lot of dependencies. I hope you may inspire from that.

[graemerocher]

We would love contributions in this area to advance this PR and make Micronaut with Kafka streams a better experience

[rgonciarz]

@graemerocher I can't promise now. I have a bigger POC project (btw. a major part might be written in Micronaut) I need to finish till autumn, but who knows after that. I'm also involved in a bigger system written from scratch using Kafka Streams mostly. At the moment I may give you a quickly you a feedback regarding documentation, especially KS once. I would add:

• more examples (e.g. with joins)
• multiple topologies (with different applicationId)
• different serialization (example with Avro or protobuf)
• how to setup security or add tracing
• health check (e.g. how to ignore false positive related with rebalancing) etc., etc.