confluent-kafka-python icon indicating copy to clipboard operation
confluent-kafka-python copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

Support for connect_cb

Open hallfox opened this issue 4 years ago • 8 comments

We have an experimental patch which adds support for setting connect_cb as a configuration, however looking back we found this issue from 2017 which says that adding it would be hard.

And it is! By doing so we discovered a deadlock that would occur since the connect_cb is called in a different thread than the other callback configs due to in part the GIL being held during calls to rd_kafka_assign. Trying to solve this, we released the GIL for the duration of rd_kafka_assign and things seem to be working well now.

That context is here because I'm curious to know:

  1. Is supporting connect_cb hard for other specific reasons?
  2. Does the assign call need to hold the GIL for other safety reasons?

Basically I'm wondering if there are some invariants I may be stepping on with these changes. Thanks.

hallfox avatar Apr 21 '21 15:04 hallfox