spring-kafka
spring-kafka copied to clipboard
spring-projects
Support for KafkaProxy for @KafkaListener
I think it would be very helpful to have support for swapping out the "backend" of @KafkaListener to support connecting via the Kafka RESTProxy instead of the normal kafka-client.
I realize you may not get 100% support, but I see MessageListenerContainer already has some UnsupportedOperationException defaulted methods, and even just very simple consuming/producing would be a nice addition.
Sounds like a separate implementation and not a mix of concerns for the @KafkaListener.
May you, please, share as much info on the matter as possible? Maybe some PoC would be really great.
I see that RESTProxy is a part of Confluent Platform: https://docs.confluent.io/current/kafka-rest/docs/index.html. We don't have any integration with that one though...
The reason it wouldn't necessarily be a mix of concerns, IMO, is that @KafkaListener could/would/should be like PreparedStatement, or JdbcTemplate, where my code doesn't care if I'm pointing at the RESTProxy or using a KafkaClient.
As it stands right now, if I develop a number of @KafkaListeners and then somebody in security says "oh, we just decided to restrict kafka--please go through the services gateway", I have a ton of work to do. If I could just include spring-kafka-rest (separate, but related), configure a RestTemplate, and bob's your uncle.
Since they are ultimately exposing the same thing via two different mechanisms, it seems like it shouldn't be that painful to do... just implement some new factories and impl classes. I'm not that familiar with the way it's been abstracted though, so it could be much worse than in #IvoryTower.
Sounds like it would just need a rest-based implementation of the KafkaMessageListenerContainer and a corresponding subclass of AbstractKafkaListenerContainerFactory to wire it into the kafka listeners.
I'd be glad to--I was just trying to get some guidance. This sounds like a good transatlantic task :) I'll try and get something next week.
Should I just do it in spring-kafka/src? or should I add (at least for the sake of reviewing) a spring-kafka-proxy/src?
I don't think a new subproject is necessary; it will share a bunch of stuff from s-k, including the abstract container. Perhaps a new package o.s.k.rest.listener. With, perhaps a rest-based KafkaTemplate in o.s.k.rest.producer or similar.
This will probably have to be a 2.2 feature since we'll need changes in ConcurrentMessageListenerContainer to support a collection of abstract containers instead of
List<KafkaMessageListenerContainer<K, V>> containers;
We'll probably fork off a 2.1.x branch after we release 2.1.5 next week and master can become the 2.2 dev branch.
Any news regarding this topic? i think many will be interested in this feature :)
We're still open to contributions, but it's unlikely to make it into 2.3; the release candidate is due in a couple of weeks.
Any updates regarding this issue ? we are interested as well to use KafkaProxy with Kafka-Spring. Thanks.
There are no updates; the project itself does not have plans to implement it; but we will consider a community contribution.