Consumer.consume with options

[hans-d]

like #41 for the producer, could there be optional parameters on consume for: partition, topic, offset and maxmessage size?

• makes consumer behave like producer and better resembles the apache kafka low level client
• allows for single connection per broker with easy consumption

[hans-d]

A possible issue with this is that Consumer is currently designed to be topic+partition specific, as the offset is kept as an object property to keep track of the current offset.

Possible solutions:

1. skip tracking the current offset, however this would brake backward compatibility
2. update the offset with the given offset (if given / should be given when topic/partition is provided) and document that keeping track off the current offset per topic per partition now becomes the responsibility of the developer.
3. move the actual connection to the broker out of this class (and reuse it), and keep the partition+topic specifics as currently.

Option 2 is for me very workable and would mean the least changes. It still provides an easy way re automatic offset tracking if used as a simple topic-partition consumer, and would allow for re-usage when used as a broker consumer.

[cainus]

Hey...Sorry about the slow response... I've been busy with other work lately and wanted to think about this for a bit too. I think you're probably right about #2. I think it could even be backward compatible if the options hash was optional.

I'd be interested in a pull request to that effect if you're offering. ;)

[hans-d]

That's ok. Would first like to check if it was something you would like to see. Will see if I can do a pull request, but that might take some time as well.