Record consumer / client ID on metrics

[mwarkentin]

We currently have a bunch of metrics in the arroyo namespace, however they are only tagged with VM / host information, rather than consumer, so it is hard (or impossible) to breakdown metrics properly per consumer.

One example of this would be to try to determine if partitions are balanced evenly across consumers. We have a metric called arroyo.consumer.partitions_assigned.count however we can only break it down by instance-id and similar tags which may include partitions assigned to multiple consumers running on the same node:

Ideally we could break this down by consumer and see if they all have the same number of partitions assigned.

[untitaker]

the min_partition tag is a similar case. we want a global tag for the entire consumer, but arroyo itself does not support that. so we had to implement it in the application.

in this particular case of checking for partition balance, I think it is sufficient to check that min == max == avg == p50 == p70 == ... of the metric, the tag breakdown is not needed IMO

[mwarkentin]

Interesting idea, but in practice if I take min - max for partition count it looks like this..