Add more options to consume

[skateinmars]

First, thanks for this project, the topics and groups commands are really impressive!

The consume command is still lacking some features though. With https://github.com/fgeller/kt/ for instance you can:

• Select which partitions to use when consuming
• Specify an offset
• Use "relative offsets" (newest-10, oldest+10)
• Select both a start and end offset to consume from

[birdayz]

i agree. so the offset can be specified with the --offset flag, but i agree, we need a bit more flexibility.

but also, i dont want to make it very complex. the reason i wrote kaf is, that also kt was a bit too complicated for my taste, especially all the flags for offsets (ofc, you can specificy the things per partitions etc) so we need a good compromise here.

a minimum for me would be a possibility to just get a single message from a topic-partition. that's sometimes a usecase when debugging/

[tzetter]

so we need a good compromise here.

I totally agree. The beauty of this tool is its simplicity / ease of use. I have been looking at different options (and played around a bit in a local branch). While kt's options are very flexible I'm not really a fan of flags like all=1-10,2=5-10. How often would anyone need to consume so specific messages anyways?

I kind of like how kafkacat does things. The possible offset flag options are:

beginning | end | stored | (absolute offset) | - (relative offset from end)

The stored option would be a separate chapter. They also have partition, count and exit flags. This is IMO quite flexible while still easy to understand.

Usage examples with this approach:

Consume the 10 last messages (on all partitions) and exit: kaf consume topic -o -10 -e

Consume the messages with offsets 10-20 on partition 1: kaf consume topic -p 1 -o 10 -c 10

Consume data on partitions 1 and 3: kaf consume topic -p 1,3

The only thing not permitted by this (that kt allows) is consuming messages with different offsets on different partitions. The results can also be a bit confusing if consuming data from multiple partitions. A warning when absolute offsets are used in combination with multiple partitions would be in order. Thoughts?

[birdayz]

i think

kaf consume topic -p 1,3

and

kaf consume topic -p 1 -o 10 -c 10

are without a doubt very reasonable.

i also agree with a flag to "exit". another option is requiring -f to follow, instead of requiring an exit flag. depending on how we do it, this may break the current behavior.