kazoo
kazoo copied to clipboard
python-zk
A zookeeper command is limited to 8KB.
When issuing a command, the output on the socket is truncated at 8KB.
zk = KazooClient()
buf = zk.command("stat")
Perhaps you could read from the socket until the socket is drained rather than truncating to only 8KB of data from the socket? https://github.com/python-zk/kazoo/blob/4d268adf9837836f05dde5ec81be0d7bbd759e78/kazoo/client.py#L653
@hannosch do you recall why you arbitrarily set it at 8KB rather than draining the socket?
IIRC 8 KB doesn't have a special meaning here. It's just a "should-be-big-enough" default value.
After reading the four-letter command list, I probably assumed their output would never be particular large.
For normal Zookeeper actions we have a different limit somewhere. And IIRC those are also limited by the jute.maxBuffer server setting.
I'm fine with either using what we use elsewhere or just draining the socket.
I stumbled upon the same issue today, as I was trying to parse the cons output, and was missing some data.
To bypass the 8192 char limit, I overrode my client command method with
def command(self, cmd='ruok'):
"""Sends a commmand to the ZK node.
Overrides methode defined at
https://github.com/python-zk/kazoo/blob/release/2.4/kazoo/client.py#L637
as it could leave some data unread from the socket.
"""
if not self._live.is_set():
raise ConnectionLoss("No connection to server")
out = []
peer = self._connection._socket.getpeername()
sock = self.handler.create_connection(
peer, timeout=self._session_timeout / 1000.0)
sock.sendall(cmd)
while True:
data = sock.recv(8192)
if not data:
break
out.append(data)
sock.close()
result = b''.join(out)
return result.decode('utf-8', 'replace')
For reference, I have a fairly used ZK server, and the cons output is ~45k char.
$ echo 'cons' | nc localhost 2181 | wc -c
44378
I have encountered a problem like @brouberol mentioned, I hope this can be resolved as soon as possible.
$ echo "mntr" |nc 127.0.0.1 2181 |wc -c
16801
Any chance of a fix here, or would you accept PRs for this? Seems pretty bad.
Hi,
Yes we would definitely accept PRs for this. I guess the addition of a new kwarg to https://github.com/python-zk/kazoo/blob/master/kazoo/client.py#L705 in order to make the size adjustable, with a default value to the current one, seems safe. But any better ideas are welcomed.
