Using answers file in a subdirectory is difficult

[johbo]

Describe the problem

I did try to locate the answers file in a subdirectory:

copier copy -a '.config/copier-answers.my-module.yaml' ~/w/copier-templates/my-module .

This did always render the answers file in the current folder instead of in a subfolder .config.

Template

Trying with the template from the documentation causes the issue:

{{_copier_conf.answers_file}}.jinja

While experimenting I found the following workaround which renders the file in the desired location:

{{ "." ~ _copier_conf.sep ~ _copier_conf.answers_file }}.jinja

To Reproduce

No response

Logs

Expected behavior

Based on the documentation I did assume that using a subfolder would just work with the documented template name.

Screenshots/screencasts/logs

No response

Operating system

macOS

Operating system distribution and version

macOS 14.7

Copier version

9.6.0

Python version

Installed via nix, assuming it uses Python 3.12 at the moment

Installation method

distro package

Additional context

Installed via Nix:

nix profile install nixpkgs#copier

[sisp]

Thanks for reporting this problem, @johbo! 👍

TL;DR: You're right, the current behavior is incorrect.

The documentation of the answers_file setting clearly states:

Path to a file where answers will be recorded by default. The path must be relative to the project root.

And the -a,--answers-file CLI flag help text clearly states:

Update using this path (relative to destination_path) to find the answers file

So, according to the documentation, it should be possible to pass a path relative to the destination. However, the current behavior is different; only the filename of a provided answers file path – no matter how – is used. This behavior clearly contradicts the documentation and intuition.

I think the root of the problem is the ability to place the {{ _copier_conf.answers_file }}.jinja file in any subdirectory of the Copier template, added via #927. As it turns out, the changes of that PR prevent rendering the answers file in a different subdirectory; only its rendered name can be overridden while the location corresponds to that of the {{ _copier_conf.answers_file }}.jinja file in the Copier template. According to my research, placing the {{ _copier_conf.answers_file }}.jinja file in any directory other than the Copier template's root directory isn't documented or tested behavior. Unless I've missed something, only the test added via #927 places the {{ _copier_conf.answers_file }}.jinja file in a subdirectory, but the same custom path is also passed via -a,--answers-file.

Rendering the answers file in a subdirectory by placing the {{ _copier_conf.answers_file }}.jinja file in a subdirectory doesn't play well with setting/overriding its location via the _answers_file setting in copier.yml or via the -a,--answers-file CLI flag (or equivalent API argument) because it would be intuitive (IMHO) to consider a custom path to be relative to that location which (a) requires users to know about the Copier template's tree layout and (b) would require the -a,--answers-file CLI flag (or equivalent API argument) to have different semantics depending on whether it's used with copier copy or copier {recopy,update}, as the latter requires a path relative to the subproject root because Copier has no access to the Copier template before reading the answers file.

Moving forward, I think Copier should only allow the {{ _copier_conf.answers_file }}.jinja file in the Copier template's root directory, so all of the answers file settings/overrides can be paths relative to the root directory with well-defined behavior. Also, there would only be one way for Copier template authors to change the location/name of the answers file via the _answers_file setting in copier.yml. And finally, we could easily add #983 and deprecate the {{ _copier_conf.answers_file }}.jinja file.

As the current behavior regarding the non-root directory location of the {{ _copier_conf.answers_file }}.jinja file isn't documented and tested, I'm inclined to consider a solution to this issue a fix rather than a breaking change. WDYT, @johbo @pawamoy @nilsdebruin?

[johbo]

Give I have a workaround -- I think it is best to give this enough time to clarify the "correct" approach. Did look into #927 and realize that it also tries to place the answers file inside of a folder .config.

If the answers file template would be like any other template, then both approaches would be just fine:

• place it in a subfolder and have it rendered there like any other template
• place it in the root folder and have the filename become a nested path (e.g. .config/copier-answers.yaml) and the rules of dynamic directory names apply

There is one difference: The answers file default can be configured via copier.yaml and even if I chose the default to be inside of .config/, there would be nothing wrong with a user of the template choosing a different location for the answers file, e.g. inside of .copier-files/. This use case would require the answers file to be inside of the root and a subpath to work instead of being stripped out.

From this angle I agree with your conclusion @sisp

I am not too familiar with Copier though, so I cannot really judge the impact of this. The implementation of #927 had probably a rationale behind and it would be good to give the author a chance to comment on this before taking any decisions.

[sisp]

I just noticed that our documentation even states that the answers file template must be in the template's root directory:

The file must be called exactly {{ _copier_conf.answers_file }}.jinja [...] in your template's root folder [...].

https://copier.readthedocs.io/en/stable/configuring/#the-copier-answersyml-file

The problem with allowing an answers file path relative to the subdirectory where the {{ _copier_conf.answers_file }}.jinja file is located is: There would be a semantic inconsistency between copier copy and copier {recopy,update} where copier copy would use a relative path to that subdirectory (e.g., .config/) and copier {recopy,update} must use a relative path to the destination path in order to be able to find the existing answers file, as the location of the answers file in the subdirectory (e.g., .config/) cannot be known by copier {recopy,update} before fetching the template, which requires the answers file – chicken and egg problem.

But I agree we shouldn't rush things and wait for some feedback also from @nilsdebruin, who implemented #927.

[sisp]

As I needed to dig into the sources already to debug the problem, I've gone ahead and created #2186 which fixes the problem according to the described plan above. But let's wait for some feedback.