Variables Megathread

[pd93]

Description

Now that the ENV_PRECEDENCE experiment is stable, it's about time we do a review of variables and environment variables and their behavior in Task. There are countless issues related to variables in v3 and we're going to be breaking them in the next major release anyway. This is the perfect time to have a think about whether there is anything else that we want to change.

This thread is a rollup issue (megathread, epic, whatever you want to call it) to discuss all variable-related ideas at a high level (no implementation details) so that we can discuss them together and in the context of a breaking change. Individual changes will be linked as child issues.

Here is a (non-exhaustive) list of the discussions that have been had elsewhere

• #482
• #993
• #1006
• #1030
• #1065
• #1276
• #1630
• #2108

[WinkelCode]

Something that would be very useful to me personally would be ways to pass information out of a task and/or command. For example, I really like the GitHub Actions approach to this: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/passing-information-between-jobs

Basically, they have a special file that key=value information can be written to, which then becomes a map that can be accessed in subsequent steps. For Task, I imagine this could be done via a single global variable or more granular, by introducing a name: or id: attribute for individual cmds.

On the topic of GitHub Actions-type functionality, I also want to mention conditional task execution based on variables, though this can already be accomplished by using templating in the task name which if it's empty, no task will run.

There are also two issues I want to mention here, both of which are about variable evaluation order becoming unexpectedly non-deterministic in some circumstances: https://github.com/go-task/task/issues/2058 (with included taskfiles) and https://github.com/go-task/task/issues/1847 (with .env files). Based on the other issues linked here, I suspect this will probably be addressed, but I still wanted to bring it up directly.

Edit: Passing information from a previous command combined with conditionally running a subsequent one (say with a command-level if: attribute) would be incredibly useful, in my opinion. I also too often up with small scripts in |- blocks because passing information isn't possible between steps.

[nhairs-lumin]

Posting some feedback here as although there are many more specific env / vars discussion threads, this is the megathread.

I'm a new user to Task. On the surface it looks pretty straight forward to use, but in practice the current state of vars and env is completely doing my head in to the point where I'm probably going to stop attempting to my existing shell scripts until a new version is dropped.

I'll detail my use case in case it helps.

Current Tools

I currently use a very heavily parametrized bash file to act as a template that I can drop into my various repositories. A public version of it can be found here. This is a sample, I have a customised version that I use for my business that does things like interacting with different AWS accounts.

Most parameters are generated from the project and but basically never change, though some are more configurable (e.g. parameters to do with AWS ECR depend on which environment I'm working with).

These parameters get used in all kinds of location, including being written to temporary dotenv files so that others tools (e.g. docker compose) can ingest them.

There are currently ~8 repositories that all work this way.

Taskfile Usage

There's a lot of things from Task to love here:

• The include system
• Passing variables and default variables

Misc

While I'm here, this link has a pretty good explainer on how vars / env passing works (at least in the current version 3). Leaving it here for other new people struggling.

[clintmod]

Here's what ChatGPT had to say when I asked it about the vars problems:

Thought for 9 seconds

In digging through the go-task repo’s issue tracker, a few recurring themes pop up around how Task handles its vars vs. env values and the order in which they’re applied:

1. Backwards Precedence (Issue #1038)

   • By default, go-task lets lower-level definitions override higher-level ones (“bottom-up”), so an environment variable you export on the shell can silently overwrite a vars: entry deeper in the Taskfile, and even two vars: entries will clobber each other depending on include order ([GitHub]1).
   • In contrast, classic tools like Make and Ant enforce immutable, top-down precedence: the first assignment “wins,” and later attempts to reassign are either ignored or error out.

2. Unexpected Mixing of env and vars (Issue #1038 & #2162)

   • Task’s separate vars: vs. env: namespaces, combined with their mutable resolution, leads to confusion: sometimes you expect env to override, other times you don’t ([GitHub]1, [GitHub]2).
   • There’s even a request to add a flag to ignore shell ENV entirely for variable resolution (#2193), which underscores how valuable predictability would be ([GitHub]2).

---

Recommended Solution: Adopt Immutable, Top-Down Variable Semantics

1. One-Pass, Write-Once Resolution

   • During Taskfile parsing, collect all vars: and env: declarations in the order they appear.
   • Once set, never allow a later declaration—whether from an include, a deeper task, or the user’s shell—to overwrite an existing value.

2. Explicit Override Only via CLI Flags

   • Mirror other build systems: let command-line flags (e.g. task --var FOO=bar) be the only way to override values, and require users to opt in.
   • Treat shell-exported ENV as read-only inputs rather than silent overrides, or provide a --ignore-env switch to disable them altogether.

3. Lazy (Deferred) Evaluation for Dynamic Needs

   • For cases where you do need “run-time” values (e.g. the output of a shell command), introduce a lazy: variable type (see #1240) that only expands at execution time—without allowing it to clobber your statically declared constants ([GitHub]3).

4. Scoping & Namespacing

   • Introduce per-task scopes or namespaces so that variables local to a task can’t leak or be mutated globally (see work-in-progress #1030). This avoids accidental cross-task pollution.

5. Fail Fast on Reassign

   • If someone does try to redefine a vars: key a second time, Task should error out rather than silently honor the newer value. That immediately surfaces conflicts.

---

Why This Helps

• Determinism: Every run sees the same values at each step—no more “it worked locally but failed in CI.”
• Reproducibility: Remote caches, parallel execution, and includes all become safe once nobody can sneak in a late-bound override.
• Easier Debugging: When a variable is immutable, you trace its origin to exactly one line in one file.
• User Expectation Alignment: Go-task users coming from Make/Ant/Gradle will find it familiar: “first definition wins, and you know where that was.”

Implementing these changes would bring go-task’s variables into line with the immutability best practices used by established build tools—and save the maintainers (and end users) from an endless stream of “why did my VAR change?” tickets.

[joeblew999]

Here's what ChatGPT had to say when I asked it about the vars problems:

Thought for 9 seconds

In digging through the go-task repo’s issue tracker, a few recurring themes pop up around how Task handles its vars vs. env values and the order in which they’re applied:

1. Backwards Precedence (Issue Environment variables from shell takes precedence over task file definition #1038)

   • By default, go-task lets lower-level definitions override higher-level ones (“bottom-up”), so an environment variable you export on the shell can silently overwrite a vars: entry deeper in the Taskfile, and even two vars: entries will clobber each other depending on include order ([GitHub]1).
   • In contrast, classic tools like Make and Ant enforce immutable, top-down precedence: the first assignment “wins,” and later attempts to reassign are either ignored or error out.

2. Unexpected Mixing of env and vars (Issue Environment variables from shell takes precedence over task file definition #1038 & Environment variables wrongly lead the precedence #2162)

   • Task’s separate vars: vs. env: namespaces, combined with their mutable resolution, leads to confusion: sometimes you expect env to override, other times you don’t ([GitHub]1, [GitHub]2).
   • There’s even a request to add a flag to ignore shell ENV entirely for variable resolution (fatal error: concurrent map read and map write #2193), which underscores how valuable predictability would be ([GitHub]2).

Recommended Solution: Adopt Immutable, Top-Down Variable Semantics

1. One-Pass, Write-Once Resolution

   • During Taskfile parsing, collect all vars: and env: declarations in the order they appear.
   • Once set, never allow a later declaration—whether from an include, a deeper task, or the user’s shell—to overwrite an existing value.

2. Explicit Override Only via CLI Flags

   • Mirror other build systems: let command-line flags (e.g. task --var FOO=bar) be the only way to override values, and require users to opt in.
   • Treat shell-exported ENV as read-only inputs rather than silent overrides, or provide a --ignore-env switch to disable them altogether.

3. Lazy (Deferred) Evaluation for Dynamic Needs

   • For cases where you do need “run-time” values (e.g. the output of a shell command), introduce a lazy: variable type (see Lazy global dynamic variable #1240) that only expands at execution time—without allowing it to clobber your statically declared constants ([GitHub]3).

4. Scoping & Namespacing

   • Introduce per-task scopes or namespaces so that variables local to a task can’t leak or be mutated globally (see work-in-progress Taskfile Scoped Variables #1030). This avoids accidental cross-task pollution.

5. Fail Fast on Reassign

   • If someone does try to redefine a vars: key a second time, Task should error out rather than silently honor the newer value. That immediately surfaces conflicts.

Why This Helps

• Determinism: Every run sees the same values at each step—no more “it worked locally but failed in CI.”
• Reproducibility: Remote caches, parallel execution, and includes all become safe once nobody can sneak in a late-bound override.
• Easier Debugging: When a variable is immutable, you trace its origin to exactly one line in one file.
• User Expectation Alignment: Go-task users coming from Make/Ant/Gradle will find it familiar: “first definition wins, and you know where that was.”

Implementing these changes would bring go-task’s variables into line with the immutability best practices used by established build tools—and save the maintainers (and end users) from an endless stream of “why did my VAR change?” tickets.

I got to admit, it took me a fair bit of time to understand how variables and envs work then using many task file includes to do overrides etc. I kind of like the idea of simple determinism and no changes at runtime.

Curious why this got a Down vote. Just want to understand the perspectives...

[trulede]

@pd93 @andreynering Regarding vars and envars, are you interested in suggestions for non-breaking changes to address this area in Task? I could share a unified approach, here or in a linked issue, that would work using v3 schema techniques.

[solvingj]

No updates from maintainers in 6 months. Pinging @pd93 for any sort of status.

[pd93]

@solvingj There is nothing further to share on this topic at this time. As soon as there is, rest assured it will be posted here. Please understand that we are volunteers working on free software with many open issues and feature requests. We all have full time jobs and limited time to work on this project and other things have taken priority recently.

I'm still very open to improvements in this area, but we need constructive comments from the community on how to proceed. Some of the comments above that express opinions or share related information are really helpful and I encourage more of these. Anyone who has thoughts on this topic is welcome to post them here and I promise that they will all be read and considered by the maintainers. However, we cannot make any promises about timelines.

[solvingj]

@pd93 Thanks for the response and contributions. Not asking for promises on timelines.

we need constructive comments from the community

Here's mine. For big breaking change like this, projects can often get paralyzed by community disagreement. However, in this case, I've read through all the comments and threads and think you are quite fortunate in this regard. While there's some nuance, the community seems quite unanimous on a few perspectives:

1. The existing behavior wasn't clearly documented behavior to be relied on
2. The existing behavior is dubious, unintuitive, and it's reasonable to describe it as a bug
3. Lots of prior art to suggest Immutable, deterministic ordering for the core behavior
4. You have an active community willing to test candidates and provide feedback

Also, the variable behavior (and implementation) is so fundamental it likely has many unforeseeable consequences, perhaps this is also paralyzing. While the notion: "If we're breaking something lets break it all and get it right once", makes sense, tactically speaking I recommend starting a public beta branch. There, begin with just one change, the most fundamental high priority one, and then iterate from there based on feedback (one change at a time).

Perhaps the most fundamental problem to start with is: "shell can take precedence over vars/env", but it's up to you where to start.

Good luck and thanks for the contribution.

[twelvelabs]

I'd like to vote for adding this issue to the list: https://github.com/go-task/task/issues/2079

I would very much like the ability to prompt for missing vars. Currently I have to rely on gum, which may or may not be installed. Would be much more portable if that was just built in to task.