fix: if self._async_conn is None raise AttributeError

[fanjindong]

if self._async_conn is None raise AttributeError: 'NoneType' object has no attribute 'cursor'

Peewee-async used in our production environment, When high concurrency, the following error is raised

AttributeError: 'NoneType' object has no attribute 'cursor'
  File "sanic/app.py", line 603, in handle_request
    response = await response
  File "utils/decorators.py", line 133, in decorated_function
    response = await f(request, *args, **kwargs)
  File "utils/decorators.py", line 237, in decorated_function
    response = await f(request, *args, **kwargs)
  File "handler/user.py", line 942, in get_users_info
    users = await mysql.execute(User.select().where(User.id.in_(otheruid)))
  File "site-packages/peewee_async.py", line 258, in execute
    return (await execute(query))
  File "site-packages/peewee_async.py", line 418, in execute
    return (await coroutine(query))
  File "site-packages/peewee_async.py", line 556, in select
    cursor = await _execute_query_async(query)
  File "site-packages/peewee_async.py", line 1446, in _execute_query_async
    return (await _run_sql(database, *query.sql()))
  File "site-packages/peewee_async.py", line 1426, in _run_sql
    cursor = await database.cursor_async()
  File "site-packages/peewee_async.py", line 854, in cursor_async
    return (await self._async_conn.cursor(conn=conn))

So I made a suggestion to fix it.

[rudyryk]

Hello @fanjindong! Thank you for reporting and PR, I still don't understand this issue. Guess we need some specific tests here to better understand conditions. From what you're saying I see that under high concurrency cursor can't be acquired. So we need to understand how to properly handle that. Could you explain your approach?

[fanjindong]

@rudyryk peewee_async.py + 843

    async def cursor_async(self):
        """Acquire async cursor.
        """
        await self.connect_async(loop=self._loop)

        if self.transaction_depth_async() > 0:
            conn = self.transaction_conn_async()
        else:
            conn = None
        try:
            return (await self._async_conn.cursor(conn=conn))
        except:
            await self.close_async()
            raise

    async def close_async(self):
        """Close async connection.
        """
        if self._async_wait:
            await self._async_wait
        if self._async_conn:
            conn = self._async_conn
            self._async_conn = None
            self._async_wait = None
            self._task_data = None
            await conn.close()

if return (await self._async_conn.cursor(conn=conn)) raise any Exception, it will be executed await self.close_async() . then self._async_conn = None.

So under high concurrency, the code seems to be stuck in a loop that causes the following error AttributeError: 'NoneType' object has no attribute 'cursor'

[rudyryk]

We need a policy for such situations. What is expected behaviour?

Current implementation assumes that if we failed to acquire cursor, something is wrong with connection and it should be re-created.

Also exception is re-raised:

# ...
except:
    await self.close_async()
    raise

So, most likely we get some internal database exception here. I'm trying to get an idea of your approach but need more explanation here.

[rudyryk]

Looking at the whole method:

    async def cursor_async(self):
        """Acquire async cursor.
        """
        await self.connect_async(loop=self._loop)

        if self.transaction_depth_async() > 0:
            conn = self.transaction_conn_async()
        else:
            conn = None

        try:
            return (await self._async_conn.cursor(conn=conn))
        except:
            await self.close_async()
            raise

Assume we have 2 concurrent requests for cursor_async():

• both of them passed await self.connect_async
• first of them failed and closed connection
• second failed at await self._async_conn.cursor with Attribute error

Yes, this is unexpected. This is important piece of code and we need to work out better and consistent approach here.

[fanjindong]

Yes, so in a short period of time, it caused tens of thousands of mistakes. Scared me to death. Minimum expected behavior: Wrong single mysql request does not affect requests in other parallels

[rudyryk]

@fanjindong The difficulty here is to distinguish between two situations:

1. Connection is overloaded by many requests
2. Connection is corrupted and needs to be re-initiated

Both scenarios will end up with exception at line (await self._async_conn.cursor(conn=conn)).

BTW, did you try to increase connection pool size?

[fanjindong]

goodbye, we have given up peewee-async

[rudyryk]

@fanjindong no problem, it's provided as-is. But I'd keep the issue opened, probably someone will find info here useful.

[kalombos]

It should be fixed since the lock has been added here We are on testing stage.