kafka4s icon indicating copy to clipboard operation
kafka4s copied to clipboard

KAYAK-1398 Shift to blocking pool inside async callback caller

Open amohrland opened this issue 3 years ago • 1 comments

Related to KAYAK-1398 and #514. Shift to blocking pool inside async ~callback~ callback caller (is there a name for this?). Untested, hence draft; will want to revisit. The motivation is the following paragraph in the KafkaProducer docs:

The buffer.memory controls the total amount of memory available to the producer for buffering. If records are sent faster than they can be transmitted to the server then this buffer space will be exhausted. When the buffer space is exhausted additional send calls will block. The threshold for time to block is determined by max.block.ms after which it throws a TimeoutException.

IOW, it appears that the enqueuing send call itself will block the underlying thread if the producer's buffer is full.

amohrland avatar Oct 12 '21 15:10 amohrland

I think the semantics of this is correct: we want to use the blocking pool to enqueue the request and to register the continuation we're passed, and I think that's what this will do.

amohrland avatar Oct 12 '21 15:10 amohrland

This is obsolete - the version in master handles wrapping correctly. In each case it's submitting an async callback so the work is already thread-shifted onto kafka's blocking pool

Daenyth avatar Jun 12 '23 13:06 Daenyth