copier-org
_external_data not working for "copier update"
Describe the problem
When updating (copier update ....) a template having _external_data defined, the files are not loaded, and the following warning is printed MissingFileWarning: File not found; returning empty dict: <file name>.
This is because the dst_path variable passed to the load_answersfile_data def, points to a temporary directory where the template is being written to (eg. /tmp/copier._main.old_copy....), and the files referenced are missing (because the file are available in the final/original working directory).
Template
Anything with _external_data
To Reproduce
- Run copier template containing
_external_data - Try to update the subproject:
copier update .
Logs
/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_user_data.py:529: MissingFileWarning: File not found; returning empty dict: source-system.yaml
warnings.warn(
Traceback (most recent call last):
File "/usr/local/py-utils/bin//copier", line 8, in <module>
sys.exit(CopierApp.run())
^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/plumbum/cli/application.py", line 640, in run
inst, retcode = subapp.run(argv, exit=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/plumbum/cli/application.py", line 635, in run
retcode = inst.main(*tailargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_cli.py", line 426, in main
return _handle_exceptions(inner)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_cli.py", line 71, in _handle_exceptions
method()
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_cli.py", line 416, in inner
with self._worker(
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 267, in __exit__
raise value
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_cli.py", line 424, in inner
worker.run_update()
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 94, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 1105, in run_update
self._apply_update()
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 1129, in _apply_update
with replace(
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 267, in __exit__
raise value
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 1138, in _apply_update
old_worker.run_copy()
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 94, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 1015, in run_copy
self._ask()
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_main.py", line 577, in _ask
answer = question.parse_answer(self.answers.init[var_name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_user_data.py", line 484, in parse_answer
return self._parse_answer(answer)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_user_data.py", line 488, in _parse_answer
ans = self.cast_answer(answer)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_user_data.py", line 244, in cast_answer
type_name = self.get_type_name()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_user_data.py", line 428, in get_type_name
type_name = self.render_value(self.type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier/_user_data.py", line 472, in render_value
return template.render({**self.context, **(extra_answers or {})})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/jinja2/environment.py", line 1290, in render
ctx = self.new_context(dict(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/jinja2/environment.py", line 1388, in new_context
return new_context(
^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/jinja2/runtime.py", line 117, in new_context
return environment.context_class(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/py-utils/venvs/copier/lib/python3.12/site-packages/copier_templates_extensions/_internal/context.py", line 48, in __init__
if "_copier_conf" in parent and (context := extension_self.hook(parent)) is not None: # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/copier._vcs.clone.c3iy2tfb/extensions/context.py", line 21, in hook
KeyError: 'source_system_name_prefix'
Expected behavior
_external_data is loaded from the place relative to destination directory.
Screenshots/screencasts/logs
No response
Operating system
Linux
Operating system distribution and version
Ubuntu 24.04.2 LTS
Copier version
9.7.1
Python version
3.12.3
Installation method
pipx+pypi
Additional context
No response
Just to clarify your use case: I assume you're loading secrets or other external data files that are not Git-tracked. Correct? It seems that both external data locations – subproject destination and temporary directory – are valid, depending on the use case:
- Path relative to the subproject destination is needed when loading untracked files like secrets.
- Path relative to the temporary directory where the internal copy of the old template version is rendered when loading tracked files like Copier answers files from other templates (as exemplified in the documentation).
I'm not sure what the best solution to this problem is. Here are some ideas:
- If an external data file is not Git-tracked, we load it from the subproject destination (not a temporary path).
- Yet another scenario for https://github.com/copier-org/copier/issues/1170#issuecomment-2917409441 although I do share @yajo's concern about requiring a (slightly) non-linear Git history with my suggested approach. Any update algorithm implementation that doesn't require replaying copies would solve this problem.
WDYT, @copier-org/maintainers?
I have the "Template composition" use-case. The source-system.yaml in the logs is actually the copier-answers file for the parent template. It is part of the final repository, it is checked in and versioned.
I do not care for the files referenced by the _external_data being changed between copier updates, which could cause false-positives when generating diff. I am aware of such issues and I can handle them on my own if needed. However, I would like to have working copier migrations :-)
Given the above, the issue to me seems more straightforward though: in the _main.py, when the old_worker is created, the dst_path is replaced with old_copy / subproject_subdir (line 1145), so that there is no way for the old copy to access any of the _external_data files (because the files should be located relative to the dst_path, which is not passed to old_worker at all). Similar thing is done for new_worker (line 1215).
The solution would be to pass external_data_source_path explicitly, which should always be dst_path.
I think you're right, both use cases I had in mind should read external data files from the subproject destination and not the temporary directory.
I'm facing the same issue as @michalbachowski
A temporary work around is create new questions where the default answer is derived from the external data. Do not specify when: False, so that those values get written to the child's answers file. NOTE, that if the parent changes the value, it will not propagate to the child on an update.
A more complete explanation of a valid workaround is https://github.com/copier-org/copier/pull/2187#issuecomment-2983938618. Do that and you won't suffer this issue for now.
The real PITA is not that warning. It's the fact that every update will give you merge conflicts.
That PR, BTW, fixes exactly this issue. It had ❌ On CI but it would be simple to fix. Answering your question, @sisp, this happens even if the external file is git tracked. It happens because, while replaying, Copier currently does not know the state of the project in the last render.
It was a bit breaking because it added a new thing to the answers file, to be able to safely replay old external answers. I closed because @sisp suggested a better solution, but if that solution is gonna take a lot of time, we can reopen. Better done than perfect IMHO.
