KazooClient.server_version() thows exception with ZooKeeper 3.6.1 server

[rickrankin]

We are in the process of upgrading ZooKeeper from 3.4.6 to 3.6.1. Upon doing so, kazoo.client.KazooClient().server_version() now throws an exception when connected to a zookeeper instance running zookeeper-3.6.1. This previously returned a valid version tuple when connecting to zookeeper-3.4.6

The ZooKeeper servers have been verified to be running. Calls to other KazooClient methods are working as expected, and ZooKeeoer utilities such as zkCli.sh are working as expected. So far, only server_version() is misbehaving.

Expected Behavior

KazooClient.server_version() method returns a valid tuple containing the zookeeper server version

Actual Behavior

A KazooException is thrown (see below).

Snippet to Reproduce the Problem

Python 3.5.6 (default, Nov 16 2018, 15:50:58)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from kazoo.client import KazooClient
>>> kz = KazooClient(hosts='my-zookeeper-server:2181')
>>> kz.start()
>>> kz.server_version()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/hfs/mcp/pyvenv/lib64/python3.5/site-packages/kazoo/client.py", line 787, in server_version
    % (1 + max(0, retries)))
kazoo.exceptions.KazooException: Unable to fetch useable server version after trying 4 times
>>>

Logs with logging in DEBUG mode

N/A

Specifications

• Kazoo version: 2.8.0
• Result of pip list command:

Package              Version
-------------------- ---------
appdirs              1.4.4
Babel                2.8.0
bcrypt               3.1.7
certifi              2020.6.20
cffi                 1.14.1
chardet              3.0.4
cliff                3.1.0
cmd2                 0.8.9
colorama             0.4.3
cryptography         3.0
debtcollector        1.22.0
decorator            4.4.2
dogpile.cache        0.9.2
idna                 2.10
importlib-resources  3.0.0
iso8601              0.1.12
jmespath             0.10.0
jsonpatch            1.26
jsonpointer          2.0
kazoo                2.8.0
keystoneauth1        4.1.0
msgpack              1.0.0
munch                2.5.0
netaddr              0.8.0
netifaces            0.10.9
openstacksdk         0.47.0
os-client-config     2.1.0
os-service-types     1.7.0
osc-lib              2.0.0
oslo.i18n            3.25.1
oslo.serialization   2.29.2
oslo.utils           3.42.1
paramiko             2.7.1
pbr                  5.4.5
pip                  20.2
prettytable          0.7.2
pycparser            2.20
PyNaCl               1.4.0
pyparsing            2.4.7
pyperclip            1.8.0
python-neutronclient 6.0.0
python-novaclient    6.0.0
pytz                 2020.1
PyYAML               5.3.1
requests             2.24.0
requestsexceptions   1.4.0
setuptools           28.8.0
simplejson           3.17.2
six                  1.15.0
stevedore            1.32.0
tabulate             0.8.7
urllib3              1.25.10
wcwidth              0.2.5
wrapt                1.12.1
zipp                 1.2.0

• Zookeeper version: 3.6.1
• Zookeeper configuration: put here any useful ZK configuration (authentication, encryption, number of ZK members, number of (concurrent?) clients, Java version, krb5 version, etc.)
  • no encryption
  • 3 cluster members, all functioning normally
  • this was discovered in a test environment with only a handful of concurrent clients (<= 10).
  • java 1.8.0.252
• Python version: 3.5.6
• OS: CentOS 7

[StephenSorriaux]

Hi,

Thanks for this issue, I will try to reproduce it.

[StephenSorriaux]

This is actually due to the new 4lw.commands.whitelist configuration added in Zookeeper 3.5.3 (see the docs). By default only srvr is whitelisted, but envi is required to get the Zookeeper server version. Deploying your Zookeeper servers with 4lw.commands.whitelist=srvr, envi will fix your issue.

I guess we can try to give a little more detail when the server_version() fails (or try to parse the envi is not executed because it is not in the whitelist. response the ZK server sent)

[rickrankin]

Thank you for investigating and pointing this out. We're coming from ZooKeeper 3.4, where this configuration item didn't exist. I've verified in our setup that whitelisting envi prevents the exception from being thrown.

I agree that a better error message might have helped. I'm not sure it would be necessary to do any parsing on the underlying response message, though. Now that I see it, just forwarding it would, I think, have given me a pretty good starting point.