GH-2381 : Support kafka parallel-consumer

[chickenchickenlove]

Introduction

Hi, Spring Team. This is chickenchickenlove, huge fan of you guys. I saw this issue (Support kafka parallel-consumer), and this comment as well!

No plans currently; but contributions are welcome!

So, i'd like to implement that integrate Parallel Consumer to spring-kafka This is skeleton code.

Please take a look when you have some free time. 🙇‍♂️ Thank you for your time!

Background

• Sometimes, Confluent Parallel consumer will be more helpful than kafka consumer, because parallel consumer have good parallel processing (If you want to know more, See this).
• Thus, i think it will be good option to spring-kafka as well.

How To use

@EnableKafka
@EnableParallelConsumer
@SpringBootApplication
public class SpringkafkasampleApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringkafkasampleApplication.class, args);
	}
}

• Annotated @EnableParallelConsumer to your Spring application. then, some spring bean related with parallel consumer, will be registered to ApplicationContext

@Component
public class MyTestClass implements ParallelConsumerCallback<String, String> {

    private final List<String> topics;
    public MyTestClass(List<String> topics) {
        this.topics = topics;
    }

    @Override
    public void accept(PollContext<String, String> context) {
        context.getConsumerRecordsFlattened().forEach(kvConsumerRecord -> System.out.println(kvConsumerRecord));
    }

    @Override
    public List<String> getTopics() {
        return this.topics;
    }
}

• and then, you should implement interface ParallelConsumerCallback and register their concrete class as spring bean by annotating @Component or @Configuration.
• then, ParallelConsumerFactory will start ParallelConsumer and ParallelConsumer process records with ParallelConsumerCallback concrete class.

ETC

• How about follow the way like @KafkaListener?
  • IMHO, it is unnecessary. because AFAIK, ParallelConsumer can allow only one callback. thus, it is not suitable for ParallelConsumer.
• Can we remove ParallelConsumerImportSelector?
  • Yes. if org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration.java class support to register spring bean, ParallelConsumerImportSelector can be removed.

Update from Apr 27.

• I added some interfaces for parallelConsumer.

Thank you for your time, Again.

[pivotal-cla]

@chickenchickenlove Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-cla]

@chickenchickenlove Thank you for signing the Contributor License Agreement!

[sobychacko]

@chickenchickenlove Thank you for the PR and the kind words. Next month, we will release the first RC version of 3.2.0. Since this is an entirely new feature and involves dependencies from io.confluent (as opposed to org.apache for the regular consumer), this needs a detailed review to come up with a plan for supporting parallel consumers. I think you are in the right direction, but we will have an internal discussion on this matter. Depending on how things go, we may have to defer this feature to the 3.3.x release, but we will let you know.

[chickenchickenlove]

Hello, @sobychacko ! Thanks for your time 🙇‍♂️

I see! however, that's sounds great. 👍 If after internal discussions, spring-kafka team decides to introduce ParallelConsumer to spring-kafka, and there is anything more to do, Please let me know. I'm more than willing to help.

Also, Please let me know in case of a decision not to introduce ParallelConsumer to spring-kafka!"

Thanks for your time, again 🙇‍♂️

[tagorenathv]

Any update on this please ?

[sobychacko]

@tagorenathv We will look at this after the GA for 3.2.0 (this month) as part of the next minor version.

[BartlomiejSzczotka]

hello @sobychacko any plans to implement it in near future? Thank you in advance for info!

[sobychacko]

@BartlomiejSzczotka Yes, we will still look at this for the 3.3.0 version. It may not be possible for it to be ready in time for the upcoming M1 release, but it will likely be in the M2 timeframe. Thanks!

[chickenchickenlove]

Hi, @sobychacko ! FYI, this PR is currently a draft because we haven't reached a consensus yet. This means that it reflects only my perspective. I believe there are better ways to support ParallelConsumer, and we can find them through discussion.

Also, i made new commits for improving code quality at April 27, 28!

So, Take your times and please feel free to talk anything, any thoughs! Once we reach a consensus, i will write test codes for this feature as well!

Thanks a lot to look at this PR! 😃

[chickenchickenlove]

Hi, @sobychacko .Gently ping. 🙇‍♂️

It's been a while. Is the integration of the parallel consumer still under review? Please let me know 🙏

[sobychacko]

@chickenchickenlove For various reasons on our end, we decided to postpone this feature to either the next minor (3.4.0) or the next major release (4.0.0). Either way, we will start working on it in a few months.

[chickenchickenlove]

Thanks for checking 👍 I will be waiting for cue 😃