query_watermark_offsets for all partitions in a topic

[tzachshabtay]

Description

Hi, query_watermark_offsets requires a partition, so to get the watermark offsets for all partitions in a topic (use-case is an admin web UI) I need to call this for each partition. As query_watermark_offsets hits the broker it is rather slow. On my machine it takes ~40ms. With a 50 partitions topic it takes me ~2 seconds to get all watermarks for the topic.

Compare that with kafkajs for example, which has a fetchTopicOffsets method, that accepts a topic and returns the offsets for all partitions, for the same 50 partitions topic it returns in ~200ms.

I wish there was a version of query_watermark_offsets that gets a topic (or a list of topics/topic-partitions?) and returns all the watermarks for that topic.

Thanks.

[edenhill]

See if you can use two calls to rd_kafka_offsets_for_times() instead, one with a timestamp (input .offset field) set to 0 for the low wmark, and one with it set to MAX_INT64 for the hwmark.

[yukurt]

@edenhill When I use an input offset of LONG_MAX or LLONG_MAX to get the high watermark offset using rd_kafka_offsets_for_times(), I get an output offset of -1. So, what I did is to use an input offset of ~0 or -1. Then I get reasonable output offsets. I hope this is correct usage?

[yukurt]

@edenhill When I use an input offset of LONG_MAX or LLONG_MAX to get the high watermark offset using rd_kafka_offsets_for_times(), I get an output offset of -1. So, what I did is to use an input offset of ~0 or -1. Then I get reasonable output offsets. I hope this is correct usage?

@edenhill may I ask if this is correct usage so we can use it in production?

[danielejuan-metr]

Hi @edenhill, we're trying to get the high water mark offsets of all the partitions of a topic. Do you have any documentation on rd_kafka_offsets_for_times() in using 0 for low water mark and -1 for high watermark?

[eiSouthBoy]

See if you can use two calls to rd_kafka_offsets_for_times() instead, one with a timestamp (input .offset field) set to 0 for the low wmark, and one with it set to MAX_INT64 for the hwmark.

Do you know that how to give a timestamp to rd_kafka_offsets_for_times()?