sarama icon indicating copy to clipboard operation
sarama copied to clipboard

Question about MaxProcessingTime timeouts

Open gmaz42 opened this issue 3 years ago • 1 comments

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama Kafka Go
v1.29.1 2.6.0 1.17
Configuration

What configuration values are you using for Sarama and Kafka?

Default configuration.

Problem Description

The documentation mentions for MaxProcessingTime:

		// The maximum amount of time the consumer expects a message takes to
		// process for the user. If writing to the Messages channel takes longer
		// than this, that partition will stop fetching more messages until it
		// can proceed again.
		// Note that, since the Messages channel is buffered, the actual grace time is
		// (MaxProcessingTime * ChannelBufferSize). Defaults to 100ms.
		// If a message is not written to the Messages channel between two ticks
		// of the expiryTicker then a timeout is detected.
		// Using a ticker instead of a timer to detect timeouts should typically
		// result in many fewer calls to Timer functions which may result in a
		// significant performance improvement if many messages are being sent
		// and timeouts are infrequent.
		// The disadvantage of using a ticker instead of a timer is that
		// timeouts will be less accurate. That is, the effective timeout could
		// be between `MaxProcessingTime` and `2 * MaxProcessingTime`. For
		// example, if `MaxProcessingTime` is 100ms then a delay of 180ms
		// between two messages being sent may not be recognized as a timeout.
		MaxProcessingTime time.Duration

When this timeout is crossed (~25s with defaults) the log line

			Logger.Printf("consumer/broker/%d abandoned subscription to %s/%d because consuming was taking too long\n",
				bc.broker.ID(), child.topic, child.partition)

is logged, but processing happens normally.

Question

I cannot understand what is the downside of this timeout being triggered, nor why it exists in first place. Any chance for some clarification? I can make a PR to update the documentation afterwards.

Thanks!

gmaz42 avatar Jan 06 '22 15:01 gmaz42

I guess the purpose of abandoning the subscription is that the slow consumer must release assigned topics/partitions to allow faster consumers in a group to take over it.

arsssen avatar Feb 16 '22 09:02 arsssen

if a partition consumer consume too slow, the channel is full, no need to fetch the partition data from broker

yws-tracy avatar Mar 24 '23 11:03 yws-tracy

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

github-actions[bot] avatar Aug 19 '23 00:08 github-actions[bot]

Closing as presumed to have been answered. Please reopen if not

dnwe avatar Aug 19 '23 00:08 dnwe