feat: Add postrender tasks phase

[rcoup]

Problem

A Copier template for Java applications requires every Java file to use .jinja extensions solely to template package names from com.example.boilerplate to com.example.{{package}}. Ideally we'd have a way to keep most template files as plain .java with a fixed com.example.boilerplate package (src/{test,main}/java/com/example/boilerplate/), then execute a simple refactoring script at copy/update time to rename package directories (eg:boilerplate/ → myservice/) and bulk-replace across files (eg: com.example.boilerplate. to com.example.myservice.).

Copier's existing task system runs after the diff calculation during updates, which breaks 3-way merge semantics. The diff compares boilerplate in the template vs myservice in the user's project, making changes undetectable.

Solution

Solving this in a generic way (since directory-per-package isn't anything Java-specific), add a new Phase.POSTRENDER that executes after template rendering but before diff calculation during updates. This enables template authors to define tasks that transform rendered output without needing .jinja extensions on every file, while maintaining the 3-way merge semantics.

Key changes

• Add Phase.POSTRENDER enum to execution phases
• Templates declare tasks in copier.yml using _postrender_tasks with support for conditional execution and custom working directories
• Execute postrender tasks in copy and update flows (on old-copy, new-copy, and destination)
• Add _update_stage variable to distinguish between previous, current, and new rendering contexts during updates
• Tests and documentation

Comments/suggestions most welcome!

[rcoup]

@sisp @pawamoy hey folks, do you have any feedback on this proposal?

[sisp]

Sorry for the delay, @rcoup! :face_with_peeking_eye:

Before getting into code review: Have you considered avoiding the .jinja suffix using the _templates_suffix setting? _templates_suffix: "" turns any file into a template file without an additional suffix.

[rcoup]

Yeah, I saw that option. I'm already using a mix of verbatim files & Jinja templates by the time I have config & CI & docs and other bits. I think dropping the explicit .jinja would make it confusing for my contributors.

[sisp]

And why do you consider bulk rename/replace better (simpler, more manageable, ...?) than templating directory and package names? I'd argue that the latter is more robust / less error-prone and involves less "magic" than maintaining a bulk rename/replace script? I mean, I imagine such a script needs to walk the file tree, and the non-Jinja placeholders need to be unique to avoid false matches. :thinking:

[rcoup]

Thanks for getting back to me :-)

For my setup, there's a mix of:

• ci/config, some templated via .jinja, others copied verbatim
• a Java code package, which (simplified) looks like

src
├── main
│   ├── docker
│   ├── java
│   │   └── com
│   │       └── example
│   │           ├── boilerplate
│   │               ├── server
│   │               │   └── filters
│   │               └── tier
│   └── resources
│       └── db
│           └── migration
└── test
    ├── java
    │   └── com
    │       └── example
    │           ├── boilerplate
    │               ├── server
    │               │   └── filters
    │               └── tier
    └── resources

com/example/boilerplate/ maps to com.example.boilerplate as a java package in both src/main/ and src/test/; and each .java file within has a package com.example.boilerplate; header line at the top.

My initial implementation was to have /boilerplate/ as /{{ package_name }}/, make each .java file a .java.jinja file with just one substitution (package com.example.{{package_name}};)

While this technically works, it's unwieldy to make changes to the template package — every new .java file needs manual work to turn it into a .jinja template; syntax validation/linting 100% fails; editing any of the files requires the same re-templating work; etc.

My substitution postrender task is really simple - it renames the two boilerplate/ directories; and updates the package header lines in all .java files within them. And the 3-way merge for template updates continues to work. Adding new boilerplate source files which only require the package line is trivial, but it can still work for more complex templating needs via .java.jinja. Editing boilerplate package source files involves simply copying them back into the template from the instantiated & edited code without re-templating.

Solving it with tasks was my first idea, but loses the merge/update capability which is key. Of course, this approach is not for everyone, and it's on me to make sure my script is robust (like any tasks script). (And I'm also minimising code that's actually in com.example.boilerplate vs com.example.common_support, but that's a slightly longer effort.)

[sisp]

Thanks for elaborating on your scenario so clearly! :1st_place_medal:

I understand your point that the Java syntax will be broken by the Jinja markup and that linters etc. won't work. I might not consider it as big a problem as you, but I acknowledge that this is probably a matter of opinion and your position seems valid, too.

I've finally found some time to think about this feature request and PR more thoroughly – sorry for the delay.

I think the current behavior of tasks is (almost) what you need, as tasks are executed right after template rendering. Copier's current update algorithm generates a fresh copy in a temporary location from the old template version using a regular run_copy call and performs two more copies from the new template version (one in the real destination directory and one in a temporary location), all of which execute the tasks after rendering. But there is one problem, which I believe would also affect your suggested solution: The task execution after the copy in the real destination directory occurs in the context of the real destination directory whose file tree is unpredictable, as the project may have diverged arbitrarily from the template. So it is entirely possible that your placeholder directory exists in the project, it would be (partially) overwritten by the internal run_copy call, and your post-rendering task would illegally rename it, thereby modifying the project in an unintended way. The kind of post-rendering tasks you need work only on fresh copies from the old and new template versions in temporary locations to avoid unintended side effects due to unpredictable project file trees. I have implemented a new update algorithm in #2376 which satisfies this requirement – except there's post-update tasks execution for backwards compatibility with making a copy in the real destination directory. That said, I don't quite like these tasks running in the real destination in the middle of the update algorithm, I don't think the purpose of these tasks is well-defined there. So, perhaps we can remove these task runs in the new update algorithm and advise template authors to use a post-migration task in case they intend to run a task after an update, which has clear semantics IMO. This will be a breaking change – but releasing the new update algorithm will be a breaking change anyway, as we'll remove support for merge conflicts via .rej files and possibly introduce slightly different 3-way merge behavior due to using the git merge command instead of a sequence of low-level Git plumbing commands.

If the new update algorithm with regular tasks offers a solution for your use case, it still means you're dependent on tasks which is an unsafe feature, so template users need to pass the --trust flag. I've been thinking about a more native approach to this kind of problem inspired by middlewares in web frameworks. Perhaps Copier could offer a way to hook into the generation/rendering process, similar to copier-template-extensions but using plain Jinja, allowing to perform rewrites like file/directory renames, Jinja template post-processing, etc. This is an ad-hoc sketch of how I imagine this could look in copier.yml:

package:
  type: str

_middleware: |
  {%- if _copier.middleware.event == 'create_path' -%}
  {%- if _copier.middleware.path.match('src/*/java/com/example/boilerplate') -%}
  {{ _copier.middlware.path.parent / package }}
  {%- endif -%}
  {%- endif -%}

We'd need to define the available events, context data, and expected "return" values of such a middleware.

WDYT?

[rcoup]

@sisp thanks for your detailed reply

it is entirely possible that your placeholder directory exists in the project, it would be (partially) overwritten by the internal run_copy call, and your post-rendering task would illegally rename it, thereby modifying the project in an unintended way.

My task script checks for directories existing/not, but yes, this is a gotcha of sorts. "Don't do that" is ok for internal uses, but I agree isn't perfect.

I have implemented a new update algorithm in https://github.com/copier-org/copier/pull/2376

Interesting — I'll have a read of this and do some experimenting, then get back to you.