Partition Size / Total Size misleading on Log Compacted Topics

[JavierMonton]

Hello 👋,

Partition size and Total size are nice on regular topics but I think they can be misleading on Log Compacted Topics (cleanup.policy = compact). As the size is calculated from Last Offset - First Offset, in general, it works pretty well. Still, on log compacted topics, offsets are not consecutive and the First Offset can be older (some messages are removed from Kafka, but not in chronological order) so the size ends up being wrong in most cases.

I have a working example, where First Offset appears as 0, so the Size is equal to the Last Offset

And when you check the actual messages in a partition, you can see that offsets are not consecutive (due to the compact cleanup policy):

I have checked this topic manually and it has around 290k available messages, instead of that 765k.

I would be more than happy to open a PR to improve this, but not sure which could be the best solution as I guess there is no way of calculating the size in a log compacted topic without reading all messages.

Maybe removing Size values when the cleanup.policy is compact? or adding an informative message?

Let me know if I can help! Thanks!

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.