Quamash crashes when no network is available

[Insoleet]

Here is a nice bug to fix...

When running our app in a network namespace without internet connexion ( https://unix.stackexchange.com/questions/68956/block-network-access-of-a-process ), quamash sometimes crashes with the following error :

DEBUG:main:Exception handler executing
ERROR:main:Future exception was never retrieved
future: <Future finished exception=gaierror(-2, 'Name or service unknown')>
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/quamash/__init__.py", line 89, in run
    r = callback(*args, **kwargs)
  File "/usr/lib/python3.4/socket.py", line 533, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service unknown

[Insoleet]

So after analyzing the problem, it seems to be something deep.

This happens when aiohttp uses loop.getaddrinfo which runs in an executor : https://github.com/python/asyncio/blob/217da1db65a1eca0ea1e14362e199f048a99ec84/asyncio/base_events.py#L536

Sometimes, the future is deleted, and the exception was not catched. The exception is then handled in the exception_handler and crashes my application. It seems like the QThreadExecutor might have a wrong behaviour somehow. But I'm really bad with threads so I have trouble analyzing it more...

[harvimt]

things to try:

set the default executor to a concurrent.futures.ThreadPoolExecutor

loop.set_default_executor(concurrent.futures.ThreadPoolExecutor(10))

Modify getaddrinfo on the event loop (say by modifying QEventLoop)

def getaddrinfo(self, host, port, *, family=0, type=0, proto=0,

flags=0): future = asyncio.Future() if self._debug: result = self._getaddrinfo_debug(host, port, family, type, proto, flags) else: result = socket.getaddrinfo(host, port, family, type, proto, flags) future.set_result(result) return future

(that is to make it synchronous, but keep the same signature)

that should give you more info.

On Thu, Dec 10, 2015 at 2:16 PM, Insoleet [email protected] wrote:

So after analyzing the problem, it seems to be something deeper.

This happens when aiohttp uses loop.getaddrinfo which runs in an executor : https://github.com/python/asyncio/blob/217da1db65a1eca0ea1e14362e199f048a99ec84/asyncio/base_events.py#L536

Sometimes, the future is deleted, and the exception was not catched. The exception is then handled in the exception_handler and crashes my application. It seems like the QThreadExecutor might have a wrong behaviour somehow. But I'm really bad with threads so I have trouble analyzing it more...

— Reply to this email directly or view it on GitHub https://github.com/harvimt/quamash/issues/42#issuecomment-163765274.

[Insoleet]

Thanks.

Here is what I could find :

• With the concurrent ThreadPoolExecutor, the same crash happens
• With a synchronized getaddrinfo, I can get a bigger stack, but there is still nothing linked to a call in my code :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<__iter__() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
    server_hostname=hinfo['hostname'] if sslcontext else None)
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 569, in create_connection
    type=socket.SOCK_STREAM, proto=proto, flags=flags)
  File "/home/inso/code/quamash/quamash/__init__.py", line 481, in getaddrinfo
    result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
    transport, proto = yield from self._create_connection(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in __iter__
    resp = yield from self._coro
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
    conn = yield from self._connector.connect(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

Some questions :

• Why in the asynchronous case the gaierror does not raise the aiohttp specific exceptions ?
• Why aren't they raised from my code, but they are from nowhere in aiohttp code ?

[harvimt]

the sync version of getaddrinfo I sent you doesn't actually handle exceptions correctly:

try:

def getaddrinfo(self, host, port, *, family=0, type=0, proto=0,

flags=0): future = asyncio.Future() try: if self._debug: result = self._getaddrinfo_debug(host, port, family, type, proto, flags) else: result = socket.getaddrinfo(host, port, family, type, proto, flags) future.set_result(result) except Exception as e: future.set_exception(result) return future

On Thu, Dec 10, 2015 at 4:46 PM, Insoleet [email protected] wrote:

Thanks.

Here is what I could find :

• With the concurrent ThreadPoolExecutor, the same crash happens
• With a synchronized getaddrinfo, I can get a bigger stack, but there is still nothing linked to a call in my code :

DEBUG:main:async_exception_handler:Exception handler executing ERROR:main:async_exception_handler:Task exception was never retrieved future: <Task finished coro=<iter() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')> Traceback (most recent call last): File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection server_hostname=hinfo['hostname'] if sslcontext else None) File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 569, in create_connection type=socket.SOCK_STREAM, proto=proto, flags=flags) File "/home/inso/code/quamash/quamash/init.py", line 481, in getaddrinfo result = socket.getaddrinfo(host, port, family, type, proto, flags) socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect transport, proto = yield from self._create_connection(req) File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection (req.host, req.port, exc.strerror)) from exc aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step result = next(coro) File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in iter resp = yield from self._coro File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request conn = yield from self._connector.connect(req) File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect .format(key, exc.strerror)) from exc aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

Some questions :

• Why in the asynchronous case the gaierror does not raise the aiohttp specific exceptions ?
• Why aren't they raised from my code, but they are from nowhere in aiohttp code ?

— Reply to this email directly or view it on GitHub https://github.com/harvimt/quamash/issues/42#issuecomment-163797841.

[Insoleet]

When catching the exception and using set_exception, here is what I get :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<__iter__() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
    server_hostname=hinfo['hostname'] if sslcontext else None)
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 581, in create_connection
    infos = f1.result()
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/futures.py", line 275, in result
    raise self._exception
  File "/home/inso/code/quamash/quamash/__init__.py", line 482, in getaddrinfo
    result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
    transport, proto = yield from self._create_connection(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in __iter__
    resp = yield from self._coro
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
    conn = yield from self._connector.connect(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

As you can see, the stacktrace is still out of the code. The exception is raised after being read in f1.result().