allow optional support for matching prerelease versions in module constraints

[TimaevAlexandr]

Terraform Version

Terraform v1.11.4
on linux_amd64

Use Cases

When developing or testing modules that have prerelease versions (e.g., 2.0.0-beta.1), it is currently impossible to reference them using common version constraint operators like >=, ~>, etc., because Terraform explicitly ignores prerelease versions in these comparisons. This makes prerelease testing workflows difficult and requires manual version pinning, which gets in the way of normal upgrade/testing flows and makes prerelease testing more difficult

This behavior also mentioned in the official Terraform documentation on version constraints:

Terraform does not match pre-release versions on >, >=, <, <=, or ~> operators.

Attempted Solutions

Tried referencing a prerelease version using a standard version constraint (>=) but Terraform does not match it

Example configuration:

module "example" {
  source  = "example.com/example/example-module"
  version = ">=2.0.0-beta.1"  # Example prerelease version
}

Available versions in the registry:

1.0.0 2.0.0-beta.1

Command:

terraform init

Terraform output:

There is no available version of module
"example-module" (main.tf:3)
which matches the given version constraint. The newest available version is
1.0.0.

Only exact matching with =2.0.0-beta.1 would work

Proposal

Introduce a new environment variable (e.g. TF_ALLOW_PRERELEASE) that, when set, changes the version constraint evaluation behavior to allow prerelease versions to match comparison operators like >=, <=, ~>, etc. This would allow for safe, explicit enabling of prerelease matching without changing default behavior

For example:

export TF_ALLOW_PRERELEASE=1
terraform init

Terraform would then accept:

module "example" {
  source  = "myorg/my-module"
  version = ">= 2.0.0-beta.1"
}

And match 2.0.0-beta.1 or later versions including future prereleases, following semver comparison logic

References

Reference This behavior is ultimately enforced by the apparentlymart/go-versions library, which is used internally by Terraform for parsing and evaluating version constraints: code comments here

Note on Implementation My goal is not to request an immediate implementation, but rather to understand whether this feature is acceptable to the maintainers. If the core team believes such a change could align with Terraform's design principles, I'd be happy to explore the implications further, including potential changes in apparentlymart/go-versions

[apparentlymart]

FWIW, I don't expect that implementing this would require any change to my "go-versions" module. Terraform could already call MeetingConstraintsExact instead of MeetingConstraints in any situation where it wants to accept selecting any prerelease that otherwise meets the constraints.

[TimaevAlexandr]

@apparentlymart Thanks for your reply! it seems that some changes to the apparentlymart/go-versions might still be needed:

I opened an issue about adding a string-based wrapper for MeetingConstraintsExact here: https://github.com/apparentlymart/go-versions/issues/8

Additionally, since MeetingConstraintsExact hasn't been tested yet, I believe it’s crucial to ensure its correctness and reliability before proceeding with integration

[TimaevAlexandr]

While we’ve discussed some possible implementation details, I think it’s still important to first determine if this feature makes sense for the project

[jbardin]

I'm not sure if MeetingConstraintsExact fits most user's intention, or if we need a different definition of constraint matching. While it's true that on a timeline 2.0.0-beta1 falls before 2.0.0 and therefor could be considered to match <2.0, versions also indicate compatibility breaks, so one cannot assume that including 2.0.0-beta1 in a config which otherwise requires >=1.0.0 is even valid.

I think for most users, opting into a prerelease means that the final GA version of that prerelease must also be included in the constraints, i.e. 2.0.0-beta is not valid for >=1.0 <2.0, but is included in >=1.0 <3.0. That might even mean extending the constraint outside of the span, and allowing 2.0.0-beta if they only have ~>2.0.0.

[jbardin]

As far as UI for this goes, I don't know if environment variables are the right choice, but it does need to be more than a binary flag, because globally enabling prereleases might include any number of modules which the user does not intend to test.

[apparentlymart]

FWIW, I agree with @jbardin that this doesn't "feel right" to me, and that's a big part of why go-versions has the default behavior it does:

Semver says that prereleases have lower precedence than their corresponding final release, and that rule makes sense when you're only trying to answer "which of these two versions is newer?", but doesn't make much sense when it comes to resolving version constraints (which is not something semver says much about).

My view has been that prereleases are essentially an "alternate timeline" containing functionality that may or may not form a coherent linear history with the final releases, since typically the whole point of them is to have some window to notice and address bugs and design flaws, which sometimes includes significant changes. Under that framing, I don't think they can participate in version constraints in any meaningful way in general, because there isn't any well-understood relationship between prereleases and their neighboring final releases.

Of course it's ultimately up to the Terraform team to decide what's right for Terraform, and the above is only representing what motivated the design of my library. I'm not particularly interested in supporting prereleases as part of the linear release timeline in the upstream library beyond the support that's already there, but I think there is already sufficient support to build what was originally proposed here if you are willing to absorb a little more of the upstream code into Terraform itself, or indeed to fork the library altogether, which are both permitted by the license. But I think anything in that direction would need to be considerably more subtle than just "prereleases are treated the same as final releases", for the reasons @jbardin described.

[apparentlymart]

Back when I wrote go-versions I did some research into how other similar systems were handling prereleases and adopted something similar, but I just noticed that Rust's Cargo has a different compromise that might work for Terraform too:

Cargo allows “newer” pre-releases to be used automatically. For example, if 1.0.0-beta is published, then a requirement foo = "1.0.0-alpha" will allow updating to the beta version. Note that this only works on the same release version, foo = "1.0.0-alpha" will not allow updating to foo = "1.0.1-alpha" or foo = "1.0.1-beta".

Cargo will also upgrade automatically to semver-compatible released versions from prereleases. The requirement foo = "1.0.0-alpha" will allow updating to foo = "1.0.0" as well as foo = "1.2.0".

(The last part of this includes some details that are specific to Cargo's syntax where it treats a version number without an operator roughly the same as go-versions/Terraform treats the ~> operator, so for the sake of interpreting this for Terraform we should read it as if all of these examples start with ~>.)

The way I might map this to Terraform is that the version constraint >= 1.1.0-beta.2 would match all of the following versions:

• 1.1.0-beta.2
• 1.1.0-beta.3
• 1.1.0-rc.1
• 1.1.0
• 1.1.1
• 1.2.0
• 2.0.0
• 2.1.0

...but it would not match the following versions:

• 1.0.0-alpha.1 (lower precedence than 1.1.0-beta.2)
• 1.0.0-beta.1 (lower precedence than 1.1.0-beta.2)
• 1.0.1-beta.1 (is a prerelease for a different base version than 1.1.0)
• 1.1.0-beta.1 (is a prerelease for a different base version than 1.1.0)
• 2.0.0-beta.1 (is a prerelease for a different base version than 1.1.0)

To put it in go-version's terms, this rule extends the "selected versions" concept to allow non-exact matching within the prereleases of whatever base version the version constraint is associated with. For example, with a version set built from the constraint I used in the above example, calling Set.Requests with version 1.1.0-beta.2 would return true, whereas today it returns false.

I still hold my general concern about there being no assumption of continuity between prereleases, but the fact that this rule has been working for Cargo in the Rust community gives me more confidence that it's a reasonable compromise that is helpful more often than it's surprising. Of course, retroactively changing how go-versions works now is technically a breaking change, and I'd want to see that it's likely to be pretty impactful in terms of helping a significant number of users in order to justify the planning and communication work to navigate such a breaking change.

---

If you wanted to add this to Terraform without upstream support then I still think that's broadly possible, like this:

1. Use constraints.ParseRubyStyleMulti to translate the "Ruby-like" constraint syntax into a tree of constraints.Spec values.

2. Instead of directly using versions.MeetingConstraints or versions.MeetingConstraintsExact, walk the constraints.Spec tree directly in Terraform-specific logic and interpret the constraints.SelectionSpec components in a similar way to how Cargo does.

   Specifically, >= 1.1.0-beta.2 would map to something like:

   bound := MustParseVersion("1.1.0-beta2") 
   coreBound := MustParseVersion("1.1.0") 
   set := AtLeast(bound).Intersection(AtMost(coreBound)).Union(AtLeast(bound).WithoutUnrequestedPrereleases())
   

   That is: >= 1.1.0-beta.2, <= 1.1.0 resolved with all of the prereleases intact, union with >= 1.1.0-beta2 with unrequested prereleases removed, so that only prereleases that belong to 1.1.0 and have a prerelease part >= beta.2 are accepted, but all final releases above the bound are accepted.

However, this does admittedly require essentially replacing the MeetingConstraintsExact function with a modified copy that handles constraints.SelectionSpec quite differently. I think everything except the little constraints.VersionSpec-to-versions.Version conversion helper is doable with existing exported API, though.

[jbardin]

I can't remember what the original need for the second go-versions package was, but FWIW we wrote hashicorp/go-version to handle pre-release matching exactly as @apparentlymart described above. A constraint of >=1.0.0-a would include matching releases, with pre-releases which lexically sort >= a, and the above list of examples starting with >= 1.1.0-beta.2 all pass and fail correctly using hashicorp/go-version. That package was written to mirror the popular ruby-gems versioning system (cargo wasn't well known (or maybe even released?) at the time), so there may be some other differences still.

[apparentlymart]

My recollection is that hashicorp/go-version did not initially have any special treatment of prerelease versions at all, and so it would happily e.g. install 1.1.0-beta.1 given the version constraint >= 1.0.0, which I believe is the cause of https://github.com/hashicorp/web-unified-docs/issues/724.

We tried to rally around updating go-version but since it's a shared library there were other users of it that objected to changing it at first, and so we swapped in my library as a workaround that ended up becoming the permanent solution for the provider installer.

Eventually the same problem arose for modules too and that time we were able to get consensus about changing hashicorp/go-version to treat that situation better, and so you implemented it in https://github.com/hashicorp/go-version/pull/35, where I can see we ended up having quite a similar conversation about whose definition of prerelease matching made the most sense to adopt. (And later someone using a different product that uses go-version complained about that behavior in https://github.com/hashicorp/go-version/issues/59, which I think is further evidence that there is no single "correct answer" to this question that pleases everyone.)

Switching back to using hashicorp/go-version for the provider installer in Terraform therefore indeed another potential way to address this, but I'd caution that the behavior of the two is subtly different in other details too. Generally I think hashicorp/go-version tends to be more permissive than apparentlymart/go-versions -- I designed mine with the intent of being strict in return for giving good feedback about mistakes, while hashicorp/go-version tends to prefer to be liberal in what it accepts at the expense of giving imprecise feedback when given strange input -- so switching in that direction is probably safer than the other direction would be, but definitely worth some careful testing.

---

Incidentally, if you're going to be looking closely at how version constraints are handled in the various parts of Terraform then you might also wish to take a look at https://github.com/opentofu/opentofu/issues/2147, which concerns some related oddities in the module installer. I assume Terraform had the same oddities, given the shared history between these two projects, and I don't know if y'all have independently addressed them in the meantime. Sharing just in case it's helpful. 🤷‍♂

[jbardin]

Thank you for the extra research @apparentlymart! I didn't have the timeline quite right, but this makes more sense now how we ended up here. I'll have to check out the linked issue too 👍

[TimaevAlexandr]

Since it's difficult to choose the best technical direction at this stage, it might make sense to first focus on what the intended user experience should look like — and use that to narrow down the implementation options

as @jbardin mentioned

As far as UI for this goes, I don't know if environment variables are the right choice, but it does need to be more than a binary flag, because globally enabling prereleases might include any number of modules which the user does not intend to test.

I experimented a bit with the libraries involved and came up with two potential options that I think are worth considering:

1. Extend go-version to support opt-in prerelease matching By default, prereleases would still be ignored, but users could explicitly include them using some notation like $>=2.0.0 or >=2.0.0-0#, or any other pattern compatible with semver This might be something for a v2.0.0 of go-version if it would otherwise be a breaking change

2. Add a Terraform-level flag, such as allow_prerelease = true in the module block. This would require Terraform-specific logic that interprets version constraints using a new rule set for prereleases — possibly similar to how Cargo handles this — and filters accordingly.

---

FWIW, I'm not entirely sure how ergonomic or intuitive both approaches would be in practice, but either way, implementing it would still require logic in Terraform’s usage of go-version to handle prerelease filtering based on this concept

Let me know if this makes sense, or if you'd prefer to take a different direction(s)