copier-org
Latest copier now has recursion error
Describe the problem
I'm now getting a recursion error in scientific-python/cookie after the latest release:
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/jinja2/environment.py", line 1388, in new_context
return new_context(
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/jinja2/runtime.py", line 117, in new_context
return environment.context_class(
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/copier_templates_extensions/_internal/context.py", line 48, in __init__
if "_copier_conf" in parent and (context := extension_self.hook(parent)) is not None: # type: ignore[attr-defined]
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/tmp/copier._vcs.clone.ujdu7qtg/helpers/extensions.py", line 17, in hook
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/copier/_types.py", line 89, in __getitem__
self._done[key] = self._pending[key]()
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/copier/_main.py", line 400, in <lambda>
"answers_file": lambda: self.answers_relpath,
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/copier/_main.py", line 629, in answers_relpath
return Path(template.render(self._render_context()))
File "/Users/runner/work/cookie/cookie/.nox/compare_copier/lib/python3.9/site-packages/jinja2/environment.py", line 1290, in render
ctx = self.new_context(dict(*args, **kwargs))
<repeats till Recursion error>
If I run it manually instead of in CI, where I'm pre-loading the answers, instead I get a KeyError:
uvx --with copier-templates-extensions copier copy gh:scientific-python/cookie package --trust --vcs-ref=HEAD Mon Apr 28 10:38:15 2025
Installed 22 packages in 73ms
Traceback (most recent call last):
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/bin/copier", line 12, in <module>
sys.exit(CopierApp.run())
~~~~~~~~~~~~~^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/plumbum/cli/application.py", line 640, in run
inst, retcode = subapp.run(argv, exit=False)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/plumbum/cli/application.py", line 635, in run
retcode = inst.main(*tailargs)
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_cli.py", line 282, in main
return _handle_exceptions(inner)
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_cli.py", line 71, in _handle_exceptions
method()
~~~~~~^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_cli.py", line 273, in inner
with self._worker(
~~~~~~~~~~~~^
template_src,
^^^^^^^^^^^^^
...<3 lines>...
overwrite=self.force or self.overwrite,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
) as worker:
^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_main.py", line 267, in __exit__
raise value
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_cli.py", line 280, in inner
worker.run_copy()
~~~~~~~~~~~~~~~^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_main.py", line 94, in _wrapper
return func(*args, **kwargs)
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_main.py", line 1015, in run_copy
self._ask()
~~~~~~~~~^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_main.py", line 601, in _ask
[question.get_questionary_structure()],
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_user_data.py", line 390, in get_questionary_structure
"message": self.get_message(),
~~~~~~~~~~~~~~~~^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_user_data.py", line 365, in get_message
if rendered_help := self.render_value(self.help):
~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier/_user_data.py", line 472, in render_value
return template.render({**self.context, **(extra_answers or {})})
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/jinja2/environment.py", line 1290, in render
ctx = self.new_context(dict(*args, **kwargs))
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/jinja2/environment.py", line 1388, in new_context
return new_context(
self.environment, self.name, self.blocks, vars, shared, self.globals, locals
)
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/jinja2/runtime.py", line 117, in new_context
return environment.context_class(
~~~~~~~~~~~~~~~~~~~~~~~~~^
environment, parent, template_name, blocks, globals=globals
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/henryschreiner/.cache/uv/archive-v0/w1rn4hvLzuCz3j6fY0JZu/lib/python3.13/site-packages/copier_templates_extensions/_internal/context.py", line 48, in __init__
if "_copier_conf" in parent and (context := extension_self.hook(parent)) is not None: # type: ignore[attr-defined]
~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/private/var/folders/_8/xtbws09n017fbzdx9dmgnyyr0000gn/T/copier._vcs.clone.4_gst8jp/helpers/extensions.py", line 10, in hook
KeyError: 'project_name'
Template
https://github.com/scientific-python/cookie
To Reproduce
To see the KeyError:
uvx --with copier-templates-extensions copier copy gh:scientific-python/cookie package --trust --vcs-ref=HEAD
Or to see the RecursionError:
uvx --with copier-templates-extensions copier copy gh:scientific-python/cookie package --trust --vcs-ref=HEAD --defaults --data=project_name=cookie-hatch --data=org=org --data=backend=hatch '--data=full_name=My Name' [email protected] --data=license=BSD --data=vcs=False
Using copier==9.7.0 also fails. But copier==9.6.0 works correctly.
Logs
https://github.com/scientific-python/cookie/actions/runs/14664378902/job/41155672563?pr=582
Expected behavior
Not to break? :)
Screenshots/screencasts/logs
No response
Operating system
macOS
Operating system distribution and version
All CI OS's and local top
Copier version
7.9.1 and 7.9.0
Python version
All
Installation method
pipx+pypi
Additional context
Why is uv not listed above? ;) Over 20% of all package installations from PyPI are now through uv (as of a week ago).
Thanks for reporting this bug, @henryiii! :pray: And sorry for the inconvenience (again). 🫣 I've submitted a PR that fixes it: #2114 However, this bug reveals a more general issue with Jinja context hooks – see the lengthy PR description.
Hey @henryii, @sisp,
My understanding is that context hooks can/should now use the phase value in the given context to make sure the hooks only run when it makes sense (avoiding redundant/expensive computations). Alternatively the hooks can catch key errors. Tagging @noirbizarre who implemented phases 🚀
I suppose we could also always catch key errors in copier-templates-ewtensions, when running hooks, to emit user warnings/messages instead of failing, but I'm not sure it's the most sensible thing to do 🙂
From what I understand, I can fix one of the cases by delaying the computation:
diff --git a/helpers/extensions.py b/helpers/extensions.py
index 39d1c2a..c7ee990 100644
--- a/helpers/extensions.py
+++ b/helpers/extensions.py
@@ -3,17 +3,25 @@ import datetime
from copier_templates_extensions import ContextHook
+class SmartDict(dict):
+ def __getitem__(self, item):
+ if item == "__ci":
+ return "github" if "github.com" in self["url"] else "gitlab"
+ if item == "__project_slug":
+ return self["project_name"].lower().replace("-", "_").replace(".", "_")
+ if item == "__type":
+ return (
+ "compiled"
+ if self["backend"] in ["pybind11", "skbuild", "mesonpy", "maturin"]
+ else "pure"
+ )
+ if item == "__answers":
+ return self["_copier_conf"]["answers_file"]
+
+ return super(item)
+
+
class CookiecutterNamespace(ContextHook):
def hook(self, context):
context["__year"] = str(datetime.datetime.today().year)
- context["__project_slug"] = (
- context["project_name"].lower().replace("-", "_").replace(".", "_")
- )
- context["__type"] = (
- "compiled"
- if context["backend"] in ["pybind11", "skbuild", "mesonpy", "maturin"]
- else "pure"
- )
- context["__answers"] = context["_copier_conf"]["answers_file"]
- context["__ci"] = "github" if "github.com" in context["url"] else "gitlab"
- return {"cookiecutter": context}
+ return {"cookiecutter": SmartDict(context)}
Edit: it has to be an actual wrapper, and not make copies, but this basic idea seems to work.
Ah, I think this also works (I hit the recursion error, of course, at the end):
diff --git a/helpers/extensions.py b/helpers/extensions.py
index 39d1c2a..fd1ced4 100644
--- a/helpers/extensions.py
+++ b/helpers/extensions.py
@@ -5,6 +5,8 @@ from copier_templates_extensions import ContextHook
class CookiecutterNamespace(ContextHook):
def hook(self, context):
+ if context.get("_copier_phase", "") == "prompt":
+ return {}
context["__year"] = str(datetime.datetime.today().year)
context["__project_slug"] = (
context["project_name"].lower().replace("-", "_").replace(".", "_")
The prompt gets called every key stroke (and a large hand full of times in-between), so reducing the (minor) computation here seems nice.
Adding a print statement with the `_copier_phase`
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
🎤 The name of your project
prompt HENRYIII
prompt HENRYIII
oprompt HENRYIII
prompt HENRYIII
nprompt HENRYIII
🎤 The name of your project
one
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
🎤 The name of your (GitHub?) org
prompt HENRYIII
prompt HENRYIII
tprompt HENRYIII
prompt HENRYIII
wprompt HENRYIII
🎤 The name of your (GitHub?) org
two
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
prompt HENRYIII
🎤 The url to your GitHub or GitLab repository
https://github.com/two/oneprompt HENRYIII
I just realized the first solution solves the recursion, too, since it delays the access to after the hook. I'll probably implement that (with the second one for good measure to keep it fast). In fact, I think it works around the (now fixed) bug in 9.5.0 too.
Oh, I hit an early version of this issue there.
The phase_copier_phase is part of the answer, it allows reducing computation in extensions by only triggering them in the phase where they are meant to.
The issue identified is that the context is fully rendered each time a template or a string is rendered. Combined with the fact that some properties hides the rendering, it makes a lot of context rendering, with a loop where some part of the context actually need the context to be rendered too.
I think, some part could be rendered once and for all, and maybe a more incremental strategy is required, where instead of rendering the entire context for each string rendered, the context is just updated when required.
Now that's easy to say, but doing is not a small task. Meanwhile, making the context lazy like it's the case delay and reduce the amount of rendering and may help, but I think the long term solution should be a single context living for the entire lifecycle and being updated when necessary. It will decouple context rendering from template rendering.
My understanding is that context hooks can/should now use the
phasevalue in the given context to make sure the hooks only run when it makes sense (avoiding redundant/expensive computations).
@pawamoy Yes, exactly.
Alternatively the hooks can catch key errors. [...] I suppose we could also always catch key errors in copier-templates-ewtensions, when running hooks, to emit user warnings/messages instead of failing, but I'm not sure it's the most sensible thing to do 🙂
@pawamoy That feels too generic to me, as key errors are not necessarily related to accessing undefined context variables.
@henryiii You've come up with clever fixes. 👌 And I'm glad you can solve the error for your template users. It's a keen observation that rendering happens at every key stroke, perhaps we can optimize this in the future. In any case, I think we should avoid the recursion with the little hack in #2114.
@noirbizarre I probably agree with you that making the context stateful and updating it along the way as Copier goes through its rendering phases would be better. I can't say right now whether there are other obstacles, but it sounds like a good idea that would reduce unnecessary re-rendering. It would be great to give this a try.