schema-registry-statistics icon indicating copy to clipboard operation
schema-registry-statistics copied to clipboard

Consume to the current "latest" or support consumption timeout

Open whatsupbros opened this issue 1 year ago • 0 comments

Is your proposal related to a problem?

Currently, looks like the tool is consuming messages continiuosly from the specified topic, and finishes only on receiving SIGINT (^C).

This leads to the situation, when it is not clear if the tool finished consumption of all currently existing data from the topic or not.

Now, to be able to understand that, one should rely on Broker throughput metrics / Control Center consumption info / kafka-consumer-groups CLI tool output for the used group ID.

Neither of these approaches is easy or obvious.

Describe the solution you'd like

In most cases continious consumption from the specified topic is not needed.

Since the tool is intended to be used for the currently existing in the topic data analysis, it would make sense to stop consumption after the current latest offset in it (the latest at the moment of the start of consumption).

The current behavior with continious consumption could be switched on with --continiously (or similar) configuration property (disabled by default).

Describe alternatives you've considered

Alternatively, a --timeout-ms (or similar) configuration property could be used (as implemented for kafka-avro-console-consumer).

When there are no more messages in the topic and the specified timeout is reached without any new message, then the tool can exit gracefully.

Proposed default value: 10000 (10 seconds) or lower

Additional context

schema-registry-statistics version: 1.2.0

whatsupbros avatar Mar 06 '23 17:03 whatsupbros