aiormq icon indicating copy to clipboard operation
aiormq copied to clipboard

aiormq 6.2.3 fails to clean up connection if first frame_receiver.get_frame() fails

Open TBBle opened this issue 2 years ago • 0 comments

We have a test in our test suite using aio-pika 7.1.2 and hence aiormq 6.2.3, which feeds an expired SSL certificate into the connect request.

The trace for the (expected) exception is:

ERROR    handler.client_service:client_service.py:664 [SSL: SSLV3_ALERT_CERTIFICATE_EXPIRED] sslv3 alert certificate expired (_ssl.c:2633)
Traceback (most recent call last):
  File "C:\Projects\PS\services\projects\handler\src\handler\client_service.py", line 648, in serve_forever
    async with self.message_handler as message_source:
  File "C:\Projects\PS\services\projects\handler\src\handler\video_generation_message_source\rabbitmq.py", line 54, in __aenter__
    await self.__listener.connect()
  File "C:\Projects\PS\services\lib\shared\src\ps\shared\rabbitmq.py", line 95, in connect
    self.connection = await aio_pika.connect_robust(rmq_url)
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\robust_connection.py", line 325, in connect_robust
    await connection.connect(
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\robust_connection.py", line 133, in connect
    await super().connect(
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\connection.py", line 135, in connect
    await self._make_connection(timeout=timeout, **kwargs)
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\connection.py", line 114, in _make_connection
    connection: aiormq.abc.AbstractConnection = await asyncio.wait_for(
  File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 442, in wait_for
    return await fut
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\connection.py", line 731, in connect
    await connection.connect(client_properties or {})
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\base.py", line 162, in wrap
    return await self.create_task(func(self, *args, **kwargs))
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\abc.py", line 40, in __inner
    return await self.task
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\connection.py", line 397, in connect
    _, _, frame = await frame_receiver.get_frame()
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\connection.py", line 143, in get_frame
    frame_header = await countdown(self.reader.readexactly(1))
  File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 479, in wait_for
    return fut.result()
  File "C:\Program Files\Python39\lib\asyncio\streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "C:\Program Files\Python39\lib\asyncio\streams.py", line 517, in _wait_for_data
    await self._waiter
  File "C:\Program Files\Python39\lib\asyncio\sslproto.py", line 528, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "C:\Program Files\Python39\lib\asyncio\sslproto.py", line 199, in feed_ssldata
    chunk = self._sslobj.read(self.max_size)
  File "C:\Program Files\Python39\lib\ssl.py", line 888, in read
    v = self._sslobj.read(len)
ssl.SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_EXPIRED] sslv3 alert certificate expired (_ssl.c:2633)

However, observed on Python 3.9 on Windows, the exception raised at aiormq\connection.py line 397 occurs after the asyncio.open_connection call on line 380, but outside the big try-catch on lines 408-449 that ensures that any exception raised will close the sockets, and hence leaks the socket without ever putting the connection somewhere we can close it (because aio_pika has wrapped this call in a factory function, so even if the best solution was to call close even on a failed connection, aio-pika would have to do that manually (as would any other user of this library))

The relevant caller in aio-pika, for reference:

    async def _make_connection(
        self, *, timeout: TimeoutType = None, **kwargs: Any
    ) -> aiormq.abc.AbstractConnection:
        connection: aiormq.abc.AbstractConnection = await asyncio.wait_for(
            aiormq.connect(self.url, **kwargs), timeout=timeout,
        )
        connection.closing.add_done_callback(
            partial(self._on_connection_close, connection),
        )
        await connection.ready()
        return connection

The symptom for us is that a later test fails with

E               ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x0000022DB33E7FA0>

because the leaked SSLProtocolTransport is not detected until tear-down of a later test (due to how pytest-aiohttp cleans up its loops, I suspect).

I found the trivial fix was to move all the code after the await asyncio.open_connection, i.e. lines 391 to 407 down inside the big try-catch, i.e. indented once and moved after line 408.

That way a failure in that block will still trigger the clean-up.

TBBle avatar Apr 11 '22 12:04 TBBle