Option to consume one message at a time? And/or configurable timeout to exit consumption?

[daluu]

This question is targeted to the MRI version, although it can be asked of JRuby as well.

Is it currently possible? From a brief look at the sample code in readme and skim of the source code, seems likely not?

The sample code shows a continuous loop with no timeout specified/specifiable. The readme portion of sample code even mentions the consume method blocks and we need to handle appropriately ourselves with some suggestions.

But looking at MRI version source code (I didn't really check on JRuby side), it looks like there is potential capability to consume a single message (and then exit)?

https://github.com/lookout/hermann/blob/master/ext/hermann/hermann_lib.c#L368

and it looks like there might be a timeout used for consumption internally?

https://github.com/lookout/hermann/blob/master/ext/hermann/hermann_lib.c#L42

It would be nice to be able to consume on a per message basis, or otherwise have a configurable timeout to force the consumption block/loop to stop as desired.

I think I recall there are Kafka libraries out there that let you consume just a single message (not necesarily Ruby, I forget which right now).

And a bit of context for my particular desired usage scenario:

I'm currently using a different Ruby library to produce (Poseidon), as having trouble with Hermann's producer, but that's a separate discussion topic. Though Hermann works better as a consumer (Poseidon had some issues we could not live with). For what I need and can do right now is produce one message at a time (using a loop if more messages need to be sent), and consume them, then process them while/after consumption.

My problem is when I just need to produce and consume a single message and then process that message when consumed (i.e. fetched), but I don't wish to do the processing within the consumer block/loop either. More like consumer block stores the fetched messages somewhere (say a global var) and then the caller that calls the consumer can process it afterwards. I could create threads to produce and consume simultaneous in this case and return to the main calling process. The producer will terminate once it's done sending (a single) message, but the consumer will block if I don't coordinate a return of control back to the main process. This is somewhat a hassle, I'd prefer to have a consumer that will terminate after consuming a single message to make it easier to coordinate the produce-consume operation on a per message basis, for however many messages I want to send. Any thoughts on how to best tackle this in the interim?

[rtyler]

This should be doable in both implementations, provided that after each message is consumed the offset is persisted.