tortoise-orm
tortoise-orm copied to clipboard
tortoise
Intermittently fail when pool release
Describe the bug
When after upgrade tortoise-orm version to 0.19.0 intermittently occur exception after update_or_create(not raw query)
Stack trace
asyncpg.exceptions._base:InterfaceError: Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7fee288912e0> 0x7fee222c5d00> is not a member of this pool
Traceback:
...
File "/x/.venv/lib/python3.9/site-packages/tortoise/models.py", line 1098, in update_or_create
File "/x/.venv/lib/python3.9/site-packages/tortoise/backends/base/client.py", line 283, in __aexit__
File "/x/.venv/lib/python3.9/site-packages/asyncpg/pool.py", line 812, in release
Additional context
When I use 0.17.7(before upgrade), never met this
I think it is related to https://github.com/tortoise/tortoise-orm/pull/1001 .. (not sure)
@long2ice on FastAPI BackgroundTasks with gunicorn worker, It happened about 1 in 20,000 times.
async def blahbalh():
await MyModel.update_or_create(status=0)
# do awaitable things, sometimes it took 5~6 minutes
# download csv from s3
# update other model (with new connection and execute raw query) << I suspect this was the problem, but not sure
await MyModel.update_or_create(status=200)
Full trace
"""
Stack trace
asyncpg.exceptions._base:InterfaceError: Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7f7a30148740> 0x7f7a300f4070> is not a member of this pool
Traceback(most recent call last):
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 202, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/workers.py", line 66, in init_process
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 142, in init_process
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/workers.py", line 84, in run
File "/.pyenv/versions/3.9.4/lib/python3.9/asyncio/runners.py", line 44, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 387, in nr_async_asgi
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/common/async_proxy.py", line 148, in next
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/common/async_proxy.py", line 120, in send
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 70, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 208, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 330, in nr_async_asgi
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/applications.py", line 112, in call
File "
@hyeongguen-song Hey, could you please paste a snapshot of the code that caused this and potentially steps to reproduce the issue ? Also, would like to know whether you're using async features of fastAPI or sync ?
@hyeongguen-song Hey, could you please paste a snapshot of the code that caused this and potentially steps to reproduce the issue ? Also, would like to know whether you're using
asyncfeatures of fastAPI orsync?
@blazing-gig Sorry for inaccurate description and code snippet🙏 I did my best to rewrite it I'm using async of FastAPI.
I can't reproduce it myself. I don't know exactly why it happens, and it doesn't happen every time.
Also getting this error. Do not know how to reproduce
Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7f6e4075fa00> 0x7f6e405b01f0> is not a member of this pool
Full stack trace:
InterfaceError: Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7f6e4075fa00> 0x7f6e405b01f0> is not a member of this pool
File "starlette/applications.py", line 124, in __call__
await self.middleware_stack(scope, receive, send)
File "starlette/middleware/errors.py", line 184, in __call__
raise exc
File "starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "starlette/middleware/sessions.py", line 86, in __call__
await self.app(scope, receive, send_wrapper)
File "starlette/middleware/base.py", line 106, in __call__
response = await self.dispatch_func(request, call_next)
File "starlette_prometheus/middleware.py", line 57, in dispatch
raise e from None
File "starlette_prometheus/middleware.py", line 53, in dispatch
response = await call_next(request)
File "starlette/middleware/base.py", line 80, in call_next
raise app_exc
File "starlette/middleware/base.py", line 69, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "starlette/middleware/cors.py", line 84, in __call__
await self.app(scope, receive, send)
File "starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "fastapi/middleware/asyncexitstack.py", line 21, in __call__
raise e
File "fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "starlette/routing.py", line 706, in __call__
await route.handle(scope, receive, send)
File "starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "starlette/routing.py", line 66, in app
response = await func(request)
File "fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
File "fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
File "broker/api/specification.py", line 138, in _handle_exceptions
result = await func(*args, **kwargs)
File "broker/api/implementation.py", line 889, in poll
worker = await _update_worker_info(report)
File "broker/api/implementation.py", line 83, in _update_worker_info
worker, _ = await models.update_or_create_worker(
File "broker/models.py", line 143, in update_or_create_worker
return await Worker.update_or_create(
File "tortoise/models.py", line 1104, in update_or_create
async with in_transaction(connection_name=db.connection_name) as connection:
File "tortoise/backends/base/client.py", line 283, in __aexit__
await self.connection._parent._pool.release(self.connection._connection)
File "asyncpg/pool.py", line 858, in release
raise exceptions.InterfaceError(
Package versions: fastapi: 0.89.1 asyncpg: 0.27.0 tortoise-orm: 0.19.3 starlette: 0.22.0 uvloop: 0.17.0
The error occurs in one of the api route handlers and is very rare.
No gunicorn is used. The server itself is executed like that in main.py
async def main():
config = uvicorn.Config(
app,
host=host,
port=port,
log_config=LOGGING_CONFIG,
)
api_server = uvicorn.Server(config)
await api_server.serve()
if __name__ == "__main__":
asyncio.run(main())
and then python main.py
In my application error occurs in this code:
return await Worker.update_or_create( # this raises InterfaceError
worker_id=worker_id,
defaults={
"strategy": strategy,
"last_polled": last_polled,
"max_instances": max_instances,
"poll_interval_seconds": poll_interval_seconds,
"terminated": terminated,
"shutting_down": shutting_down,
"hostname": hostname,
},
)
