rust-rdkafka icon indicating copy to clipboard operation
rust-rdkafka copied to clipboard

`StreamConsumer` not waking

Open trtt opened this issue 3 months ago • 1 comments

This StreamConsumer racy behavior can be observed in the recent release, probably after the move to Event-based API. Some of the events (stats, logs etc.) are processed internally and are consumed by the poll returning None. Since librdkafka only wakes the queue when it transitions from empty -> non-empty, waker might be not called at any point in the future in this case.

Event order example:

  1. new message event
  2. new stats event
  3. poll processes stats and returns None
  4. StreamConsumer sets up waker and returns Poll::Pending
  5. waker is never called by librdkafka since the queue never went empty

Probably connected issue: https://github.com/fede1024/rust-rdkafka/issues/638

trtt avatar Mar 30 '24 07:03 trtt

I sketched out a potential solution to the problem: https://github.com/fede1024/rust-rdkafka/pull/666

trtt avatar Mar 30 '24 07:03 trtt