SystemError: AST constructor recursion depth mismatch

[luhn]

After upgrading to Python 3.11 and Chameleon 4.2.0, I'm encounter errors seemingly at random when rendering my templates.

SystemError: AST constructor recursion depth mismatch (before=174, after=138)

My rendering code is pretty straightforward. Each invocation instantiates a fresh PageTemplate

def _render_html(template_name, template_data):
    """
    Load the chameleon template and render it.

    This may be blocking and should not be called in the main reactor thread.

    """
    macro = PageTemplate(_load_file('emails/template.pt'))
    template_data['template'] = macro.macros['template']
    macro = PageTemplate(_load_file('emails/dw_template.pt'))
    template_data['dw_template'] = macro.macros['template']
    template = PageTemplate(
        _load_file('emails/{}.pt'.format(template_name)),
        encoding='utf8',
    )
    return template(**template_data)

I attempted to set CHAMELEON_EAGER=1 as suggested in https://github.com/malthe/chameleon/issues/361, but the errors still persist.

[malthe]

Might have to do with some specific expression in your template. Can you try and home in on which expression causes this – assuming that this is the case, that it can be isolated to a particular syntax?

[luhn]

Unfortunately I'm just not sure where to start with that.

It doesn't happen consistently, so I can't reproduce it locally.

The traceback doesn't seem to point at any specific spot in a template:

Traceback (most recent call last):
--
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1993, in _inlineCallbacks
result = context.run(
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/src/app/souschef/models/reservation.py", line 438, in email
yield self.request.mailer.send_templated_email(
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork()  # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/src/app/souschef/utils/mailer.py", line 108, in send_templated_email
_render_html(template, data),
File "/usr/src/app/souschef/utils/mailer.py", line 263, in _render_html
macro = PageTemplate(_load_file('emails/template.pt'))
File "/usr/local/lib/python3.11/site-packages/chameleon/zpt/template.py", line 225, in __init__
super().__init__(body, **config)
File "/usr/local/lib/python3.11/site-packages/chameleon/template.py", line 138, in __init__
self.write(body)
File "/usr/local/lib/python3.11/site-packages/chameleon/template.py", line 241, in write
self.cook(body)
File "/usr/local/lib/python3.11/site-packages/chameleon/template.py", line 168, in cook
program = self._cook(body, digest, names)
File "/usr/local/lib/python3.11/site-packages/chameleon/template.py", line 251, in _cook
source = self._compile(body, builtins)
File "/usr/local/lib/python3.11/site-packages/chameleon/template.py", line 284, in _compile
compiler = Compiler(
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1001, in __init__
module.body += self.visit(node)
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1050, in visit
iterator = visitor(node)
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1094, in visit_Module
program = self.visit(node.program)
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1050, in visit
iterator = visitor(node)
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1113, in visit_MacroProgram
stmts = self.visit(macro)
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1052, in visit
for key, group in itertools.groupby(
File "/usr/local/lib/python3.11/site-packages/chameleon/compiler.py", line 1141, in visit_Macro
body += emit_func_convert_and_escape("__quote")
File "/usr/local/lib/python3.11/site-packages/chameleon/codegen.py", line 68, in wrapper
expr = parse(textwrap.dedent(source), mode=mode)
File "/usr/local/lib/python3.11/site-packages/chameleon/astutil.py", line 45, in parse
return compile(source, '', mode, ast.PyCF_ONLY_AST)
SystemError: AST constructor recursion depth mismatch (before=84, after=119)

Maybe it's a race condition? Should I try wrapping the render function in a mutex?

[luhn]

Worth noting: I'm seeing tracebacks from all three PageTemplate calls in the function, so doesn't seem to be specific to any one template.

[malthe]

If you can share some version of your template, that would be quite helpful – or if that is a problem, you can try and do something akin to a bisection, cutting away half of the template until it compiles. This should help you identify exactly the expression that is causing the problem.

Or you could set up a breakpoint here:

File "/usr/local/lib/python3.11/site-packages/chameleon/codegen.py", line 68, in wrapper

That seems to be the spot where you could find the culprit.

[luhn]

It doesn't happen consistently. Most of the time, the template will compile successfully, but a handful of times the exact same template will fail to compile.

So far I'm unable to reproduce it locally. And I can't deploy those debugging steps into production.

I wrapped the whole function in a mutex last night and so far no more errors, so it does seem to be a thread safety issue.

[malthe]

Maybe same issue as #361.

[luhn]

I tried CHAMELEON_EAGER as suggested in #361 and that didn't work, but I'm now realizing that's moot if I'm reinstantiating the template every call.

Was able to reproduce locally:

from threading import Thread
from chameleon import PageTemplate
import traceback
import pdb


template = """<html>
	<body>
		Hello!
	</body>
</html>
"""


running = True


def run():
    global running
    while running:
        try:
            PageTemplate(template)
        except Exception:
            if running:
                running = False
                # pdb.post_mortem()
                traceback.print_exc()


t1 = Thread(target=run)
t1.start()
t2 = Thread(target=run)
t2.start()
t1.join()
t2.join()

[malthe]

This seems to be related to https://github.com/python/cpython/issues/106905 – for now, I think we have to accept that Python 3.11.5 is not compatible with Chameleon.

[malthe]

The fixed has been merged for 3.13, available in 3.13.0-alpha.2, but it still being backported for 3.11 and 3.12.