Unable to reference modules with slash in ref name

[denis256]

Terraform Version

Terraform v1.7.0
on linux_amd64

Terraform Configuration Files

module "test" {
  source = "github.com/denis256/terraform-test-module.git//modules/test-file?ref=team/ABC-1234-component"
}

Debug Output

https://gist.github.com/denis256/ca6eac9fe36b5ee6f27c67db6f1ca79d

Expected Behavior

Module source to be downloaded from git branch with slashes

Actual Behavior

Path github.com/denis256/terraform-test-module.git//modules/test-file?ref=team/ABC-1234-component is evaluated as git::https://github.com/denis256/terraform-test-module.git//ABC-1234-component/modules/test-file?ref=team" which lead to error.

2024-01-22T16:52:28.884Z [TRACE] ModuleInstaller: test source address has changed from "git::https://github.com/denis256/terraform-test-module.git//modules/test-file?ref=master" to "git::https://github.com/denis256/terraform-test-module.git//ABC-1234-component/modules/test-file?ref=team"
2024-01-22T16:52:28.884Z [TRACE] ModuleInstaller: discarding previous record of test prior to reinstall
2024-01-22T16:52:28.884Z [TRACE] ModuleInstaller: cleaning directory .terraform/modules/test prior to install of test
2024-01-22T16:52:28.888Z [TRACE] ModuleInstaller: test address "git::https://github.com/denis256/terraform-test-module.git//ABC-1234-component/modules/test-file?ref=team" will be handled by go-getter
Downloading git::https://github.com/denis256/terraform-test-module.git?ref=team for test...
2024-01-22T16:52:28.888Z [TRACE] getmodules: fetching "git::https://github.com/denis256/terraform-test-module.git?ref=team" to ".terraform/modules/test"
2024-01-22T16:52:29.417Z [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

Steps to Reproduce

1. terraform init

Additional Context

No response

References

No response

[apparentlymart]

Thanks for reporting this, @denis256!

This source address seems to be relying on the github.com shorthand, which causes some extra preprocessing to turn it into a Git URL for Terraform to use. Although it should work as you described (and I'd still like to fix it), I wonder if skipping that shorthand would be a viable workaround in the meantime:

  source = "git::https://github.com/denis256/terraform-test-module.git//modules/test-file?ref=team/ABC-1234-component"

or, if you prefer to use SSH instead:

  source = "git::ssh://[email protected]/denis256/terraform-test-module.git//modules/test-file?ref=team/ABC-1234-component"

I don't actually know if this will make a difference, but if it does turn out to work when written explicitly in this way then that would be a good clue as to where the bug is.

Terraform actually delegates much of this behavior to an upstream library called go-getter, and so it's likely that this will need to be fixed there rather than here, but we'll need to do some further debugging first to confirm. Thanks again!

[apparentlymart]

Due to us having delegated much of this work to go-getter, there aren't a lot of detailed unit tests for this behavior in Terraform itself. I wrote a quick temporary test in internal/getmodules to see if this would reproduce easily:

func TestSplitPackageSubdir(t *testing.T) {
	tests := []struct {
		Input string
		Want  [2]string
	}{
		{
			"github.com/denis256/terraform-test-module.git//modules/test-file?ref=team/ABC-1234-component",
			[2]string{"github.com/denis256/terraform-test-module.git?ref=team/ABC-1234-component", "modules/test-file"},
		},
	}

	for _, test := range tests {
		t.Run(test.Input, func(t *testing.T) {
			pkg, sub := SplitPackageSubdir(test.Input)
			got := [2]string{pkg, sub}
			want := test.Want
			if got != want {
				t.Errorf("wrong result\ninput: %s\ngot:  %#v\nwant: %#v", test.Input, got, want)
			}
		})
	}
}

Unfortunately(?), this test actually passed as written!

--- PASS: TestSplitPackageSubdir (0.00s)
    --- PASS: TestSplitPackageSubdir/github.com/denis256/terraform-test-module.git//modules/test-file?ref=team/ABC-1234-component (0.00s)

...so it seems like the problem is more subtle here than just the subpath splitting logic not working right. I would guess that this problem arises in some later step after we've already performed this pkg vs. subpath split.

---

If this does turn out to be a go-getter bug and we send a fix upstream, it's possible we'll also need to make the same fix to the go-slug "sourceaddrs" package, since it aims to be backward-compatible with a subset of what Terraform accepts as source addresses today and its parsing and normalization logic was based on that from go-getter, so it might have inherited the same bug.

[mrachuta]

Hello, for quick workaround you can use "/" as hex code %2F. It's working for me:

17:16:57  - main.gcp_app_network_module in .terraform/modules/main.gcp_app_network_module/modules/gcp-network-module
17:16:57  Downloading git::https://github.com/mrachuta/terraform-resources.git?ref=feature%2Fadd-ssl-to-cloudsql-module for main.gcp_cloudsql_module...