parallel-consumer icon indicating copy to clipboard operation
parallel-consumer copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

Integration with Kafka Connect

Open kondamudikarthik opened this issue 3 years ago • 4 comments

This is more of a question, can we integrate parallel-consumer into KafkaConnect? Has anyone tried it? I believe we might need to change pollConsumer method in WorkerSinkTask to read messages in parallel. Is this something in your roadmap? Thanks!

Hi @astubbs, do you have any thoughts on this?

kondamudikarthik avatar Jun 22 '21 05:06 kondamudikarthik

Hi! Yes, I've thought about this many times, and brought it up myself. I'm not aware of anyone trying it yet - but it seems like a very obvious marriage of performance :) But no, it's not planned to implement due to lack of demand.

astubbs avatar Jul 27 '21 14:07 astubbs

@astubbs Thanks for getting back, I'm willing to do this as we use connect a lot in our data pipelines. I will explore around on this and get back to you.

kondamudikarthik avatar Jul 27 '21 18:07 kondamudikarthik

Consider the inverse - teach PC to run connectors...

astubbs avatar May 17 '22 17:05 astubbs

I have started experimenting on this now. Sources or sinks in particular? I have working hacks with both connect inside PC, and PC inside connect. But I think connect sinks inside PC is the one that has some chance of being published. Would that be useful?

astubbs avatar Jul 15 '22 12:07 astubbs

Closing Issue

johnbyrnejb avatar Jul 07 '23 14:07 johnbyrnejb