Module installer incorrectly handles truncated pessimistic version constraints

[ericrichtert]

Terraform Version

λ terraform version
Terraform v1.0.11
on windows_amd64
+ provider registry.terraform.io/hashicorp/aws v2.70.0

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 2"
    }
  }
  required_version = ">= 1"
}


module "mft-s3eventlambda" {
  source  = "xxx/xxxx/terraform-mft-s3eventlambda/aws"
  version = "~> 10"
}

Expected Behavior

Available:

• AWS provider 3.66.0 and 2.70.0
• terraform-mft-s3eventlambda module 10.10.28 and 11.10.10

We expect AWS provider version 3.66.0 and module version 11.10.10 (according to https://www.terraform.io/docs/language/expressions/version-constraints.html)

Actual Behavior

Downloading xxx/xxxx/terraform-mft-s3eventlambda/aws 11.10.10 for mft-s3eventlambda...
- mft-s3eventlambda in .terraform\modules\mft-s3eventlambda

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 2.0"...
- Installing hashicorp/aws v2.70.0...
- Installed hashicorp/aws v2.70.0 (signed by HashiCorp)

Steps to Reproduce

terraform init -backend=false

[alisdair]

Hi @ericrichtert, thanks for reporting this.

For the provider constraint, Terraform is behaving as designed here. A constraint of "~> 2" is interpreted as "~> 2.0" (as you can see from the logs), and from the docs:

~>: Allows only the rightmost version component to increment. For example, to allow new patch releases within a specific minor release, use the full version number: ~> 1.0.4 will allow installation of 1.0.5 and 1.0.10 but not 1.1.0. This is usually called the pessimistic constraint operator.

So "~> 2.0" will allow any 2.x.y version, but not anything greater than or equal to 3.0.0.

What's less clear is why the module version you're seeing is not behaving the same way. Terraform only supports version constraints for registry modules. Is the installation source a private module registry?

If you're able to provide debug logs for the terraform init run which installs these modules (removing any sensitive information such as hostnames) that would help us figure out what's happening here. Running the command with the environment variable TF_LOG=trace would help. Thanks!

[ericrichtert]

@alisdair How can I (secure) provide the logs? We use a privaye registry and don't want all the logs in github

[alisdair]

You could either redact any sensitive information before posting, or encrypt the logs using our current PGP public key, found at https://www.hashicorp.com/security

[ericrichtert]

2021-11-26T16:39:41.491+0100 [DEBUG] Adding temp file log sink: C:\XXXX\terraform-log903565567 2021-11-26T16:39:41.492+0100 [INFO] Terraform version: 1.0.11 2021-11-26T16:39:41.492+0100 [INFO] Go runtime version: go1.16.4 2021-11-26T16:39:41.492+0100 [INFO] CLI args: []string{"C:\XXXX\terraform.exe", "init"} 2021-11-26T16:39:41.493+0100 [TRACE] Stdout is a terminal of width 120 2021-11-26T16:39:41.493+0100 [TRACE] Stderr is a terminal of width 120 2021-11-26T16:39:41.493+0100 [TRACE] Stdin is a terminal 2021-11-26T16:39:41.498+0100 [DEBUG] Attempting to open CLI config file: C:\XXXX\terraform.rc 2021-11-26T16:39:41.498+0100 [INFO] Loading CLI configuration from C:\XXXX\terraform.rc 2021-11-26T16:39:41.502+0100 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins 2021-11-26T16:39:41.502+0100 [DEBUG] ignoring non-existing provider search directory C:\XXXX\terraform.d\plugins 2021-11-26T16:39:41.504+0100 [DEBUG] ignoring non-existing provider search directory C:\XXXXCorp\Terraform\plugins 2021-11-26T16:39:41.504+0100 [INFO] CLI command args: []string{"init"} 2021-11-26T16:39:41.508+0100 [TRACE] ModuleInstaller: installing child modules for . into .terraform\modules 2021-11-26T16:39:41.509+0100 [DEBUG] Module installer: begin mft-s3eventlambda 2021-11-26T16:39:41.509+0100 [TRACE] ModuleInstaller: mft-s3eventlambda is not yet installed 2021-11-26T16:39:41.509+0100 [TRACE] ModuleInstaller: cleaning directory .terraform\modules\mft-s3eventlambda prior to install of mft-s3eventlambda 2021-11-26T16:39:41.509+0100 [TRACE] ModuleInstaller: mft-s3eventlambda is a registry module at XXXX/XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:41.509+0100 [DEBUG] mft-s3eventlambda listing available versions of XXXX/XXXX/terraform-mft-s3eventlambda/aws at XXXX 2021-11-26T16:39:41.509+0100 [DEBUG] Service discovery for XXXX at https://XXXX/.well-known/terraform.json 2021-11-26T16:39:41.509+0100 [TRACE] HTTP client GET request to https://XXXX/.well-known/terraform.json 2021-11-26T16:39:41.778+0100 [DEBUG] fetching module versions from "https://XXXX/v1/modules/XXXX/terraform-mft-s3eventlambda/aws/versions" 2021-11-26T16:39:41.778+0100 [DEBUG] GET https://XXXX/v1/modules/XXXX/terraform-mft-s3eventlambda/aws/versions 2021-11-26T16:39:41.778+0100 [TRACE] HTTP client GET request to https://XXXX/v1/modules/XXXX/terraform-mft-s3eventlambda/aws/versions 2021-11-26T16:39:42.238+0100 [DEBUG] found available version "10.10.21" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "11.10.10" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.19" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.17" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.25" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.10" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.12" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.27" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.18" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.28" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.20" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.26" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.22" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.13" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.23" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.24" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.16" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.14" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.11" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] found available version "10.10.15" for XXXX/terraform-mft-s3eventlambda/aws 2021-11-26T16:39:42.239+0100 [DEBUG] looking up module location from "https://XXXX/v1/modules/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/download" 2021-11-26T16:39:42.239+0100 [DEBUG] GET https://XXXX/v1/modules/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/download 2021-11-26T16:39:42.239+0100 [TRACE] HTTP client GET request to https://XXXX/v1/modules/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/download 2021-11-26T16:39:42.267+0100 [TRACE] ModuleInstaller: mft-s3eventlambda XXXX/XXXX/terraform-mft-s3eventlambda/aws 11.10.10 is available at "https://XXXX/v1/modules/tarball/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/module.tar.gz" 2021-11-26T16:39:42.267+0100 [DEBUG] will download "https://XXXX/v1/modules/tarball/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/module.tar.gz" to .terraform\modules\mft-s3eventlambda 2021-11-26T16:39:42.267+0100 [TRACE] fetching "https://XXXX/v1/modules/tarball/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/module.tar.gz" to ".terraform\modules\mft-s3eventlambda" 2021-11-26T16:39:57.140+0100 [TRACE] ModuleInstaller: mft-s3eventlambda "https://XXXX/v1/modules/tarball/XXXX/terraform-mft-s3eventlambda/aws/11.10.10/module.tar.gz" was downloaded to .terraform\modules\mft-s3eventlambda 2021-11-26T16:39:57.140+0100 [TRACE] ModuleInstaller: mft-s3eventlambda should now be at .terraform\modules\mft-s3eventlambda 2021-11-26T16:39:57.150+0100 [DEBUG] Module installer: mft-s3eventlambda installed at .terraform\modules\mft-s3eventlambda

[alisdair]

Thanks, that log was very helpful! I've been able to reproduce this against the public registry with the following configuration:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "~> 2"

  name = "my-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = true

  tags = {
    Terraform = "true"
    Environment = "dev"
  }
}

This results in installing version 3.11.0 of this module, instead of the latest 2.x.y version. As far as I can tell this appears to be a bug in Terraform, which I'll take a look at shortly.

As a workaround, I recommend specifying the two-digit version with the pessimistic operator (version = "~> 10.0" for your case). In my testing this behaved as expected, installing the latest version with the same major number.

[alisdair]

After some further investigation, I understand where the issue lies now, but the fix isn't trivial.

The module installer is using the hashicorp/go-version library to parse and check version constraints, whereas the newer provider installer is using apparentlymart/go-versions. These differ in how they handle pessimistic version constraints like `"~> 10".

The go-version library's interpretation of "~> 10" is fairly useless, as it is equivalent to "> 10.0.0". I think the best fix here would be to rework the module installer to use the go-versions library instead, making the behaviour consistent across both providers and modules.

[ericrichtert]

@alisdair What is the status of this issue?

[crw]

Hi @ericrichtert, due to the work involved in making this change, it is not currently planned. It is in the triage list to be considered for a future release.

In the meantime, please use the workaround described in https://github.com/hashicorp/terraform/issues/30025#issuecomment-980153015:

As a workaround, I recommend specifying the two-digit version with the pessimistic operator (version = "~> 10.0" for your case). In my testing this behaved as expected, installing the latest version with the same major number.

Thanks very much for the report and your patience!

[ericrichtert]

@crw @alisdair any update on this issue?

[crw]

No updates to report.