pykafka
pykafka copied to clipboard
Error committing offsets for topic ,pykafka.exceptions.UnknownMemberId
consumer Heartbeat worker exit because of SocketDisconnectedError Then it cause the error committing offsets(errors: {<class 'pykafka.exceptions.UnknownMemberId'>
Result: consumer is still consume, but offset not be committed
pykafka==2.8.0
broker version=0.10.0
Python 2.7.5
consumer_timeout_ms=500
auto_offset_reset=OffsetType.LATEST
auto_commit_enable=True
auto_commit_interval_ms=5000
other parameter is default
2019-04-05 22:13:26,599 {producer.py-_send_request:482} DEBUG Sending 6 messages to broker 10069
2019-04-05 22:13:26,609 {producer.py-_send_request:482} DEBUG Sending 6 messages to broker 10073
2019-04-05 22:13:26,638 {producer.py-_send_request:537} DEBUG Successfully sent 6/6 messages to broker 10069
2019-04-05 22:13:26,654 {producer.py-flush:730} DEBUG max_request_size reached. producing batch
2019-04-05 22:13:26,654 {producer.py-_send_request:482} DEBUG Sending 6 messages to broker 10074
2019-04-05 22:13:26,655 {producer.py-_send_request:537} DEBUG Successfully sent 6/6 messages to broker 10073
2019-04-05 22:13:26,699 {producer.py-_send_request:537} DEBUG Successfully sent 6/6 messages to broker 10074
2019-04-05 22:13:26,950 {simpleconsumer.py-_auto_commit:522} DEBUG Autocommitting consumer offset for consumer group cg_des_raw_log_process_3 and topic tp_des_raw_log
2019-04-05 22:13:26,950 {simpleconsumer.py-commit_offsets:557} DEBUG Committing offsets for 5 partitions to broker id 221338468
2019-04-05 22:13:28,095 {handlers.py-worker:219} INFO my debug:exception=Traceback (most recent call last):
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/handlers.py", line 215, in worker
res = shared.connection.response()
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/connection.py", line 226, in response
raise SocketDisconnectedError("<broker {}:{}>".format(self.host, self.port))
SocketDisconnectedError: <broker xxxxx:13014>
2019-04-05 22:13:28,098 {simpleconsumer.py-fetch:810} INFO Updating cluster in response to error in fetch() for broker id -568108188
2019-04-05 22:13:28,100 {cluster.py-update:509} DEBUG Updating cluster, attempt 1/3
2019-04-05 22:13:28,102 {connection.py-connect:164} DEBUG Connecting to xxxxxx:13014
2019-04-05 22:13:28,111 {connection.py-connect:174} DEBUG Successfully connected to xxxxx:13014
2019-04-05 22:13:28,139 {handlers.py-stop:190} INFO RequestHandler.stop: about to flush requests queue
2019-04-05 22:13:28,140 {cluster.py-_update_brokers:383} INFO Discovered 3 brokers
2019-04-05 22:13:28,140 {cluster.py-_update_brokers:407} INFO Broker 100.91.36.76:13014 metadata unchanged. Continuing.
2019-04-05 22:13:28,141 {cluster.py-_update_brokers:397} INFO Reconnecting to broker id -568108188: xxxxx:13014
2019-04-05 22:13:28,142 {connection.py-connect:164} DEBUG Connecting to xxxxxx:13014
2019-04-05 22:13:28,144 {connection.py-connect:174} DEBUG Successfully connected to xxxxxx:13014
2019-04-05 22:13:28,145 {handlers.py-stop:190} INFO RequestHandler.stop: about to flush requests queue
2019-04-05 22:13:28,148 {cluster.py-_update_brokers:407} INFO Broker 100.91.49.13:13014 metadata unchanged. Continuing.
2019-04-05 22:13:28,148 {cluster.py-_update_topics:134} INFO Discovered 13 topics
................
2019-04-05 22:14:12,038 {simpleconsumer.py-commit_offsets:557} DEBUG Committing offsets for 5 partitions to broker id 221338468
2019-04-05 22:14:12,091 {producer.py-_send_request:482} DEBUG Sending 15 messages to broker 10074
2019-04-05 22:14:12,129 {producer.py-_send_request:537} DEBUG Successfully sent 15/15 messages to broker 10074
2019-04-05 22:14:32,882 {simpleconsumer.py-_handle_success:775} DEBUG Fetched 1 messages for partition 1
2019-04-05 22:14:32,882 {simpleconsumer.py-_handle_success:779} DEBUG Partition 1 queue holds 632 messages
2019-04-05 22:14:32,883 {simpleconsumer.py-_handle_success:775} DEBUG Fetched 461 messages for partition 4
2019-04-05 22:14:33,028 {managedbalancedconsumer.py-fetcher:250} INFO Sending heartbeat from consumer '10.114.8.4:18267-221338468-12482047'
2019-04-05 22:14:33,031 {handlers.py-worker:219} INFO my debug:exception=Traceback (most recent call last):
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/handlers.py", line 213, in worker
shared.connection.request(task.request)
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/connection.py", line 206, in request
raise SocketDisconnectedError("<broker {}:{}>".format(self.host, self.port))
SocketDisconnectedError: <broker 100.91.49.13:13014>
2019-04-05 22:14:33,033 {managedbalancedconsumer.py-fetcher:268} INFO my debug:exception traceback=Traceback (most recent call last):
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/managedbalancedconsumer.py", line 254, in fetcher
self._consumer_id)
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/broker.py", line 542, in heartbeat
return future.get(HeartbeatResponse)
File "/usr/local/services/beetle2-1.0/thirdparty/lib/python2.7/site-packages/pykafka/handlers.py", line 74, in get
raise self.error
SocketDisconnectedError: <broker xxxx>