asyncpg icon indicating copy to clipboard operation
asyncpg copied to clipboard

Asyncpg does not seem to clear memory allocated to Pool object after db.disconnect() (possible memory leak)

Open IvanovCosmin opened this issue 2 years ago • 1 comments

  • asyncpg version: 0.25.0 (tried with 0.23.0 and 0.24.0)
  • PostgreSQL version: 11.16
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce the issue with a local PostgreSQL install?: No, using the postgres:11.16 docker image directrly
  • Python version: 3.10.5 (+ latest 3.9 and 3.8)
  • Platform: Ubuntu 64 and docker
  • Do you use pgbouncer?: No
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?: N/A
  • Can the issue be reproduced under both asyncio and uvloop?: I only tried asyncio

We observed PODs running out of memory. They are reading from a queue constantly and calling this function

async def process_request(message, db : ):
    await db.connect()
    # do_something(message, db)
    await db.disconnect()

I am confident this is related to asyncpg as here are some of the objects that are created in memory with every iteration of connect/disconnect.

  asyncpg.pgproto.pgproto.ReadBuffer |        1000 |    132.81 KB
             collections.OrderedDict |        1000 |    125.00 KB
   asyncpg.pool.PoolConnectionHolder |        1000 |    117.19 KB
          asyncio.events.TimerHandle |        1001 |    109.48 KB

I have tried:

  • waiting for the TimerHandle to expire (60s)
  • calling engine.dispose() manually
  • calling the garbage collector manually

Sometime in the past we encountered a problem that made 10000s of connections in a few seconds (crashing the DB) that was fixed by abandoning this connect/disconnect pattern, so maybe these two are related.

The db is initialised as such

databases.Database(db_uri)

This problem does not exists in the sqlite driver.

IvanovCosmin avatar Jun 23 '22 09:06 IvanovCosmin

Can you reproduce this by using just asyncpg? Otherwise it isn't clear if this isn't a bug in databases.

elprans avatar Jun 23 '22 18:06 elprans