Request to add more Consumer Metrics

[sayan1886]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
1.32.0	2.3.0	1.16

Configuration

What configuration values are you using for Sarama and Kafka?

	km.saramaConfig = sarama.NewConfig()
	km.saramaConfig.Version = sarama.V2_3_0_0
	km.saramaConfig.Net.SASL.Enable = true
	km.saramaConfig.Net.SASL.User = kCfg.SaslUsername
	km.saramaConfig.Net.SASL.Password = kCfg.SaslPassword
	km.saramaConfig.Net.TLS.Enable = true
	km.saramaConfig.Net.TLS.Config = &tls.Config{RootCAs: certPool, MinVersion: tls.VersionTLS12}
	if verbose {
		sarama.Logger = &log.Logger{}
	}
	km.saramaConfig.ClientID = "go-kafka-metrices"
	km.saramaConfig.Consumer.Return.Errors = true

	// Sarama Producer Policy
	// Asynchronous producers are not recommended to enable both Errors and Successes,
	// generally just enable Errors. Synchronous producers must all be turned on,
	// because the success or failure of the transmission will be returned synchronously
	km.saramaConfig.Producer.Return.Errors = true    // Set whether to return an error message
	km.saramaConfig.Producer.Return.Successes = true // Set whether to return success information

	// Partition allocation policy
	switch kCfg.Assignor {
	case "sticky":
		km.saramaConfig.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategySticky
	case "roundrobin":
		km.saramaConfig.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategyRoundRobin
	case "range":
		km.saramaConfig.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategyRange
	default:
		log.Panicf("Unrecognized consumer group partition assignor: %s", kCfg.Assignor)
	}
	// Offset
	km.saramaConfig.Consumer.Offsets.Initial = sarama.OffsetNewest
	// channel length
	km.saramaConfig.ChannelBufferSize = kCfg.ChannelBufferSize

IBM Kafka

Logs

When filing an issue please provide logs from Sarama and Kafka if at all possible. You can set sarama.Logger to a log.Logger to capture Sarama debug output.

logs: CLICK ME

Problem Description

Request for adding more metrics related to Consumers

	Metric Name Key (Type) (Unit)	Description
1	consumer assigned partitions kafka.consumer.partitions.assigned (double gauge)	The number of partitions currently assigned to consumer
2	consumer fetcher max lag kafka.consumer.fetcher.max.lag (double gauge)	Max lag in messages per topic partition
3	consumer fetcher avg lag kafka.consumer.fetcher.avg.lag (double gauge)	Average lag in messages per topic partition
4	consumer fetcher lag kafka.consumer.fetcher.lag (double gauge)	Lag in messages per topic partition
5	bytes consumed rate kafka.consumer.bytes.rate (double gauge) (bytes/sec)	The average number of bytes consumed per second
6	records consumed rate kafka.consumer.records.rate (double gauge) (rec/sec)	The average number of records consumed per second
7	consumer records max lag kafka.consumer.records.lag.max (double gauge)	The maximum lag in terms of number of records for any partition
8	consumer records per request kafka.consumer.requests.records.avg (double gauge) (rec/req)	The average number of records per request
9	consumer fetch rate kafka.consumer.fetch.rate (double gauge) (fetches/sec)	The number of fetch requests per second
10	consumer fetch avg size kafka.consumer.fetch.size (double gauge) (bytes)	The average number of bytes fetched per request
11	consumer fetch max size kafka.consumer.fetch.size.max (double gauge) (bytes)	The maximum number of bytes fetched per request
12	consumer fetch latency kafka.consumer.fetch.latency (double gauge) (ms)	The average time taken for a fetch request
13	consumer fetch max latency kafka.consumer.fetch.latency.max (double gauge) (ms)	The maximum time taken for a fetch request
14	consumer throttle time kafka.consumer.throttle.time (double gauge) (ms)	The avarage throttle time in ms
15	consumer throttle max time kafka.consumer.throttle.time.max (double gauge) (ms)	The max throttle time in ms
16	consumer node requests rate kafka.consumer.node.request.rate (double gauge) (req/sec)	The average number of requests sent per second.
17	consumer node request size kafka.consumer.node.request.size (double gauge) (bytes)	The average size of all requests in the window.
18	consumer node in bytes rate kafka.consumer.node.in.bytes.rate (double gauge) (bytes/sec)	Bytes/second read off socket
19	consumer node request max size kafka.consumer.node.request.size.max (double gauge) (bytes)	The maximum size of any request sent in the window.
20	consumer node out bytes rate kafka.consumer.node.out.bytes.rate (double gauge) (bytes/sec)	The average number of outgoing bytes sent per second to servers.
21	consumer node request max latency kafka.consumer.node.request.latency.max (double gauge) (ms)	The maximum request latency
22	consumer node request latency kafka.consumer.node.request.latency (double gauge) (ms)	The average request latency
23	consumer node responses rate kafka.consumer.node.response.rate (double gauge) (res/sec)	The average number of responses received per second.
24	consumer connection count kafka.consumer.connections (double gauge)	The current number of active connections.

[tomatopunk]

Hi folks, I really need these consumer metrics, is there any progress update on this issue, is anyone working on this? @dnwe

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[sayan1886]

any update on the feature request? @dnwe

[dnwe]

@sayan1886 it's definitely high up on the agenda to get at least some of these useful consumer metrics implemented, but we just haven't gotten around to starting on it yet as we've had a bunch of work to do in the protocol area, as well as general bug fixing and improvements

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[UpsideDownPicture]

@dnwe Just making sure this request is still on the radar to implement!