Burrow icon indicating copy to clipboard operation
Burrow copied to clipboard

Consumer status is unpredictable when multiple topics are consumed

Open ashi009 opened this issue 1 year ago • 0 comments

https://github.com/linkedin/Burrow/blob/4a05b203ce40d82def3af986eeca5350570e6b96/core/internal/evaluator/caching.go#L226-L252

This piece of code loops over a map of topics, and if the last topic's last partition is reporting ok, the consumer status will be ok.

Given that the map iteration in go is randomized, the consumer status is unpredictable.

The following are the real world effect from this:

  1. The metric from burrow of a consumer when scraping at 2m interval: image

  2. The metric from burrow-exporter which requests burrow at 30s interval, and then being scrapped at 2m interval: image

The more frequently we query (as burrow uses 30s cache expiration by default), the more likely to see non-OK consumer status.

ashi009 avatar Jan 09 '24 08:01 ashi009