Propan icon indicating copy to clipboard operation
Propan copied to clipboard

Feature/Question: consume a batch of messages

Open chrisgoddard opened this issue 1 year ago • 3 comments

I'm not sure if this is already possible, but I'm wondering if there is a way to consume a batch of messages off a topic (I'm working with Kafka).

I want to consume either 5 minutes of messages or 50,000 messages (whichever comes first) and then process that collection of messages as a batch.

Right now, each handler is executed once for each message so I don't see a straightforward way to pull multiple messages in one function.

Is there a way to do this currently? If not, do you think this is something that could be possible?

chrisgoddard avatar Aug 31 '23 18:08 chrisgoddard

@chrisgoddard sorry, but in the current implementation there is no way to consume batches But you can find this functionality in the V2, just wait 2 weeks for release

Lancetnik avatar Aug 31 '23 18:08 Lancetnik

It would be great to add support for Redis-Streams too.

farzbood avatar Sep 08 '23 06:09 farzbood

@chrisgoddard please, check FastStream for this feature Tomorrow will be a great release All your Propan code should works with FastStream correctly: just migrate @handle to @subscriber and change some imports paths

Lancetnik avatar Sep 17 '23 18:09 Lancetnik