Potential memory leak

[fantamiracle]

If you wait for a few minutes you can see the memory bloat.

pybar3 version: 0.9.7

from pybars import Compiler, strlist
import json
import time
compiler = Compiler()
from memory_profiler import profile

@profile
def replace_vars(mystr, variables):
    def _json(this, item):
        return strlist(json.dumps(item))
    if isinstance(mystr, str):
        try:
            template = compiler.compile(mystr)
        finally:
            pass
        return template(variables, helpers={"json": _json})
    return mystr

while True:
    replace_vars("{{json event}}", 
                 {'event': {'event': 'connectivity.state.changed', 'timestamp': 1640217025.7437212, 'connectStatusOk': False}, 'request': {'httpParams': {'body': '{{json event}}', 'headers': ['User-Agent: Engine', 'Content-Type: application/json'], 'method': 'post' }}})
    time.sleep(.1)

Any thoughts for a fix to a way to get around it?

[radove]

Did you guys ever find a resolution or workaround for this issue? We're experiencing something similar with v0.9.7

[suhjohn]

Specifically it's this code

exec(compile(container.full_code, filename, 'exec', dont_inherit=True), mod.__dict__)

In _compiler.py that seems to keep its reference.

[suhjohn]

@radove @fantamiracle This is the fix

for key in list(sys.modules.keys()): # you might want to adjust the string in startswith # to match the dynamic modules in your case if key.startswith('pybars._templates.'): print("removing module: ", key) del sys.modules[key]

The dynamically generated module when compiling the template is not de-referenced. We were compiling a new template every request on a Django (the nature of the template is dynamic because it can be user generated in our use-case so we can't define them as globals), and this was causing the templates to continue to exist even if the variable that the dict is assigned to gets dereferenced

[suhjohn]

As an addendum, as they state it, the compiler is not threadsafe, so you will need to use a lock when initializing the Compiler and compiling

lock = threading.Lock()

def compile_prompt_template(prompt: str, args: dict): with lock: try: compiler = Compiler() template = compiler.compile(prompt) except: pass

Something like the above.