ruby-kafka

ruby-kafka copied to clipboard

Kafka::AsyncProducer#produce has only one mechanism of propagating backpressure: raising Kafka::BufferOverflow. This is not a very good mechanism — it's more of a blow-up than build-up — but it does the job of notifying the caller when progress cannot be made.

However, this really falls short when it comes to relieving backpressure: providing simply no way of being notified of it. Thus, caller code is forced to periodically check the backpressure by itself. Such polling, though, as is common wisdom, is inefficient at scale.

Besides, there should be no need to poll within the same process! Dependents can simply block on their dependencies using efficient concurrency primitives provided by the base system. Not only should it be more efficient than polling, but also handle pressure-relief automatically.