client.py __del__(self) failed exception handling

[bryanhiestand]

python_etcd-0.4.3-py3.4.egg/etcd/client.py intermittently fails to catch multiple exceptions on exit. The two exceptions I was able to reproduce were:

Exception ignored in: <bound method Client.__del__ of <etcd.client.Client object at 0x7f305ea9ef98>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/python_etcd-0.4.3-py3.4.egg/etcd/client.py", line 227, in __del__
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 95, in clear
  File "/usr/lib/python3/dist-packages/urllib3/_collections.py", line 90, in clear
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 69, in <lambda>
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 451, in close
TypeError: catching classes that do not inherit from BaseException is not allowed

Exception ignored in: <bound method Client.__del__ of <etcd.client.Client object at 0x7f2fe1db9f98>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/python_etcd-0.4.3-py3.4.egg/etcd/client.py", line 227, in __del__
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 95, in clear
  File "/usr/lib/python3/dist-packages/urllib3/_collections.py", line 85, in clear
  File "/usr/lib/python3.4/_collections_abc.py", line 444, in values
TypeError: 'NoneType' object is not callable

Offending code:

def __del__(self):
    """Clean up open connections"""
    if self.http is not None:
        try:
            self.http.clear()
        except ReferenceError:
            # this may hit an already-cleared weakref
            pass

Code to reproduce this bug (repeated runs required, bug not always triggered):

#!/usr/bin/env python3

import sys
import etcd

client = etcd.Client(port=2379)

try:
    directory = client.get('/does/not/exist')
    print(directory)
except Exception:
    print('I did not want that key anyways.')

print('Exiting...')
sys.exit(0)

Changing ReferenceError to Exception resolves the issue for me, but I suspect you will want to use something else or resolve the issue elsewhere.

[carsonoid]

This is an issue for me as well when trying to use python-etcd for salt data caching.

[arizvisa]

As a temporary workaround, we can probably explicitly call Client.__del__ or something that assigns None to Client.http so that when the gc calls this method it won't try to access properties that have already been released.

I'm considering writing a context manager wrapper around the connection logic that utlizes this solution...

[cenkalti]

I am wrapping the Client class as workaround:

class Client(etcd3.Etcd3Client):

    # Workaround for https://github.com/jplana/python-etcd/issues/179
    def __del__(self):
        try:
            super().__del__()
        except Exception:
            pass