kafka-python icon indicating copy to clipboard operation
kafka-python copied to clipboard

client_async: Allow throwing an exception upon socket error during wakeup

Open shimon-armis opened this issue 2 years ago • 5 comments

Motivation

When wakeup() is called, we sometime notice that we get an endless prints: "Unable to send to wakeup socket!". Those prints are spamming the logs.

Description

This commit aims to address it by allowing restating the application via an intentional exception raise. This behavior is configurable and its default is backward compatible.

We see this issue with kafka-python 2.0.2


This change is Reviewable

shimon-armis avatar Jun 26 '23 11:06 shimon-armis

@dpkp Can u please review this PR?

shimon-armis avatar Jun 26 '23 11:06 shimon-armis

@shimon-armis do you have a traceback of the logs in question, and some sample code?

wbarnha avatar Aug 03 '23 03:08 wbarnha

@shimon-armis do you have a traceback of the logs in question, and some sample code?

Hey @wbarnha, It's been a while :) but here are some snippet I found in my slack's DM. Note, It isn't a complete traceback. W0221 08:13:16.082472 kafka.client 1-140694973044480 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:331] Node 11 connection failed -- refreshing metadata W0221 08:13:16.083114 kafka.client 1-140694973044480 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:331] Node 12 connection failed -- refreshing metadata W0221 08:13:16.084225 kafka.client 1-140694973044480 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:331] Node 13 connection failed -- refreshing metadata W0221 08:13:16.224422 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:331] Node 19 connection failed -- refreshing metadata W0221 08:13:16.254804 kafka.client 1-140694589003520 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:331] Node 20 connection failed -- refreshing metadata W0221 08:13:16.324988 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:937] Unable to send to wakeup socket! W0221 08:13:16.425574 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:937] Unable to send to wakeup socket! W0221 08:13:16.526113 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:937] Unable to send to wakeup socket! W0221 08:13:16.626606 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:937] Unable to send to wakeup socket! W0221 08:13:16.727129 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:937] Unable to send to wakeup socket! W0221 08:13:16.827740 kafka.client 1-140694981437184 /home/runner/pipeline.runfiles/pip_kafka_python/site-packages/kafka/client_async.py:937] Unable to send to wakeup socket!

shimon-armis avatar Mar 09 '24 17:03 shimon-armis

@wbarnha Note that the suggested change is an acceptable practice, under the circumstances. I probably missed your previous comment but will happy if we can get this done quickly 🙏

shimon-armis avatar Mar 09 '24 17:03 shimon-armis

I'm resuming work on this in https://github.com/wbarnha/kafka-python-ng/pull/134!

wbarnha avatar Mar 10 '24 03:03 wbarnha