Memory Leak with Lambdas/Functions in Queries

[newhouseb]

Hi! PonyORM is great, thanks for making it!

We're hitting a memory leak in our production deployments, when we use a lambda (or other def'ed function) in a query. See the below for a minimal reproduction:

import gc
from pony.orm import * 

db = Database()

class Foo(db.Entity):
    value = Required(str)

db.bind('sqlite', ':memory:')
db.generate_mapping(create_tables=True)

@db_session
def leak():
    test = lambda f: 'foo' in f.value
    list(select(f for f in Foo if test(f)))

@db_session
def dont_leak():
    list(select(f for f in Foo if 'foo' in f.value))

# This only works on Linux, but avoids any debugging overhead that would
# corrupt our measurements
def getCurrentMemoryUsage():
    ''' Memory usage in kB '''
    with open('/proc/self/status') as f:
        memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]
    return int(memusage.strip())

start = getCurrentMemoryUsage()
for i in range(100):
    leak()
    gc.collect()
end = getCurrentMemoryUsage()
print("Lambda Leaked", end - start, "kB over 100 runs")

start = getCurrentMemoryUsage()
for i in range(100):
    dont_leak()
    gc.collect()
end = getCurrentMemoryUsage()
print("Inline Leaked", end - start, "kB over 100 runs")

On PonyORM 0.7.14 this results in the following:

root@ff0bdd6af6b3:/app/backend# python ponyleak.py
Lambda Leaked 1400 kB over 100 runs
Inline Leaked 0 kB over 100 runs