Using terragrunt scaffold with GitLab URLs

[gmaghera]

Describe the bug Scaffold does not seem to work with GitLab URLs

To Reproduce

remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
WARN[0000] Failed to find last release tag for git::https://gitlab.com/rivian/dc.git
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/rivian/dc.git//platform/terraform-modules/s3.git to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'https://gitlab.com/rivian/dc.git': /usr/local/bin/git exited with 128: Cloning into '/var/folders/1q/b7d7p2g91fddxq6jwt237cdm6xcxmy/T/getter2517239202/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

or

❯ terragrunt scaffold https://gitlab.com/rivian/dc/platform//terraform-modules//s3
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
WARN[0000] Failed to find last release tag for git::https://gitlab.com/rivian/dc.git
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/rivian/dc.git//platform/terraform-modules/s3 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'https://gitlab.com/rivian/dc.git': /usr/local/bin/git exited with 128: Cloning into '/var/folders/1q/b7d7p2g91fddxq6jwt237cdm6xcxmy/T/getter1330873649/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

not applicable

Expected behavior Please provide an example for how to do this against a GitLab repository.

Nice to have

• [ ] Terminal output
• [ ] Screenshots

Versions

• Terragrunt version: 0.55.20
• Terraform version: 1.5.5
• Environment details (Ubuntu 20.04, Windows 10, etc.): Darwin 23.3.0, macOS 14.3.1 Additional context Add any other context about the problem here.

[denis256]

Hi, git ssh URLs are also not handled correctly?

[denis256]

I suspect wasn't used .git URL to clone repo In my tests for https://gitlab.com/denis256/terragrunt-tests I used:

$ terragrunt scaffold https://gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template?ref=v10.0.0 to /tmp/test1 
INFO[0004] Running boilerplate generation to /tmp/test1 
[boilerplate] 2024/03/28 20:43:07 Loading boilerplate config from /tmp/boilerplate967725454/boilerplate.yml
[boilerplate] 2024/03/28 20:43:07 Loading boilerplate config from /tmp/boilerplate967725454/boilerplate.yml
INFO[0004] Scaffolding completed

$ terragrunt scaffold gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/denis256/terragrunt-tests.git//scaffold/default-template?ref=v10.0.0 to /tmp/test1 
INFO[0004] Running boilerplate generation to /tmp/test1 
INFO[0004] /tmp/test1/terragrunt.hcl was updated        
INFO[0004] Scaffolding completed    

[lorengordon]

Using .git on the end of the clone url is something of a github thing. It's not strictly necessary, and for example, codecommit doesn't use/support it at all.

[gmaghera]

Hi @denis256, thanks for looking into this. If you look at my original quote, I did try without the .git piece terragrunt scaffold https://gitlab.com/rivian/dc/platform//terraform-modules//s3 failed.

The error suggests to me that there is an assumption about directory level. Your example apparently uses a personal GitLab project which is always two levels deep: gitlab.com/<username>/<projectname> -- and that works. Compare that to the error I am getting:

❯ terragrunt scaffold https://gitlab.com/rivian/dc/platform//terraform-modules//s3
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
WARN[0000] Failed to find last release tag for git::https://gitlab.com/rivian/dc.git
INFO[0000] Scaffolding a new Terragrunt module git::https://gitlab.com/rivian/dc.git//platform/terraform-modules/s3 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'https://gitlab.com/rivian/dc.git': /usr/local/bin/git exited with 128: Cloning into '/var/folders/1q/b7d7p2g91fddxq6jwt237cdm6xcxmy/T/getter1330873649/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/rivian/dc.git/' not found
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

The project path is https://gitlab.com/rivian/dc/platform//terraform-modules/s3, but the error say this fatal: repository 'https://gitlab.com/rivian/dc.git/' not found. "dc" is a group not a git project, and you gotta go two levels deeper before the s3 project is found.

(and I see that I used the // notation twice, by accident, but I get the same exact error with just one or none, too.)

[gmaghera]

@lorengordon I am not sure about GitHub, but with GitLab if you go into the web UI both SSH and HTTPS URLs end in .git. But they do work without it too.

[gmaghera]

Hi, git ssh URLs are also not handled correctly?

Apparently not. Or I'm not using the call signature correctly...

~/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox experiment-with-scaffold* 1h 36s
(⎈|dev-us-east-1:kube-system)❯ terragrunt scaffold [email protected]:rivian/dc/platform/terraform-modules/s3.git
INFO[0002] Scaffolding a new Terragrunt module git::ssh://[email protected]/rivian/dc/platform/terraform-modules/s3.git?ref=4.2.0 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0002] error downloading 'ssh://[email protected]/rivian/dc/platform/terraform-modules/s3.git?ref=4.2.0': /usr/local/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

~/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox experiment-with-scaffold*
(⎈|dev-us-east-1:kube-system)❯ terragrunt scaffold [email protected]:rivian/dc/platform/terraform-modules/s3
INFO[0001] Scaffolding a new Terragrunt module git::ssh://[email protected]/rivian/dc/platform/terraform-modules/s3?ref=4.2.0 to /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/scaffold-sandbox
ERRO[0001] error downloading 'ssh://[email protected]/rivian/dc/platform/terraform-modules/s3?ref=4.2.0': /usr/local/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git
ERRO[0001] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[denis256]

Hi, yes, there is an issue... happens in my tests too, after some debugging looks like it is an issue in go-getter library which is used to pull content, something similar happens in go-getter CLI too

$ go-getter "github.com/denis256/s3.git//module2" /tmp/qwe
2024/04/01 18:01:56 success!

$ go-getter "gitlab.com/denis25-tests/dc/platform/terraform-modules/s3.git//module2" /tmp/qwe
2024/04/01 18:02:19 Error downloading: error downloading 'https://gitlab.com/denis25-tests/dc.git': /usr/bin/git exited with 128: Cloning into '/tmp/getter72444115/temp'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/denis25-tests/dc.git/' not found

Will raise an issue in the library GH project

[denis256]

https://github.com/hashicorp/go-getter/issues/479

[gmaghera]

Thanks for the update @denis256.

[gmaghera]

@denis256 could you remove the "awaiting response" label? Is there something else I can provide?

[gmaghera]

@denis256 my colleague @tvizor is looking into addressing the issue with go-getter, but I think there is some ambiguity to this.

Considering this example from https://blog.gruntwork.io/introducing-terragrunt-catalog-and-scaffold-aead126dcf10:

terragrunt scaffold \
  github.com/gruntwork-io/terragrunt-infrastructure-modules-example//modules/s3-bucket

The repo URL in this case is github.com/gruntwork-io/terragrunt-infrastructure-modules-example and the module is under the subpath modules/s3-bucket.

That may be clear in the context of Terragrunt, where // means download everything under that directory, but it may not mean anything to Hashicorp's go-getter. I assume that the above example would also work without the double slash.

For a GitLab URL which can nest groups up to 20 levels deep, this presents a problem. Does Terragrunt equate //with the end of the (GitLab) repo URL and tries to use go-getter to retrieve only the part up to it? For handling both repo-only URLs and repo+subpath URLs, I think the usage of // would become mandatory.

[denis256]

Hi, I think we need to use the approach when a double slash is the separator of remaining path, similar to how it is for Terraform modules

https://developer.hashicorp.com/terraform/language/modules/sources#modules-in-package-sub-directories

FYI: @brikis98

[josh-padnick]

@yhakbar @denis256 Any chance we can make a final call on this so we can keep things moving?

[gmaghera]

Hi @josh-padnick.

My colleague, @tvizor, has submitted a PR for addressing https://github.com/hashicorp/go-getter/issues/479, opened by @denis256.

Could we work together on resolving this? Scaffold is a game-changer, in my view. I'd love to see it work (more robustly) with GitLab.

[yhakbar]

Hello! Sorry this hasn't been moving along at the rate you would prefer, @gmaghera .

Is my understanding right that the only issue here is with the underlying go-getter implementation for GitLab repositories nested under a group? If so, how can we help to resolve this?

I believe we would need your pull request to be merged by the maintainers of go-getter to have your repositories supported, right?

[gmaghera]

Hi @yhakbar -- yes, if that PR gets merged, and we get a new version of Terragrunt which includes that code, that should solve the issue.

It's not just our repositories, in its current shape go-getter and scaffold only support a fraction of GitLab. There may be a gap with testing non-free GitLab repos which can contain nested groups in their URL. And there is a request to add the same feature to GitHub... See https://github.com/orgs/community/discussions/4837

[gmaghera]

Go-getter seems abandoned, perhaps there is some lull due to the IBM/Hashicorp thing...

@yhakbar could we explore some alternate solutions? We could spin a release from the fork, or perhaps some other way to get this working. Anything we can do to get scaffold functional with GitLab Enterprise...

[lorengordon]

Does it work in terraform natively? If not, and you can easily reproduce the issue, might be worth opening an issue there. At this point, the go-getter project is primary only updated to support use cases from terraform...

[gmaghera]

Does it work in terraform natively? If not, and you can easily reproduce the issue, might be worth opening an issue there. At this point, the go-getter project is primary only updated to support use cases from terraform...

Yes, the issue is not specific to Terragrunt at all. @denis256 has already opened an issue for it, which is linked to the PR which would fix it. See https://github.com/gruntwork-io/terragrunt/issues/3031#issuecomment-2030181923

[lorengordon]

Does it work in terraform natively? If not, and you can easily reproduce the issue, might be worth opening an issue there. At this point, the go-getter project is primary only updated to support use cases from terraform...

Yes, the issue is not specific to Terragrunt at all. @denis256 has already opened an issue for it, which is linked to the PR which would fix it. See https://github.com/gruntwork-io/terragrunt/issues/3031#issuecomment-2030181923

That issue is on go-getter, not on terraform. Like I said, if the issue is not opened against terraform and confirmed there to impact terraform, no maintainer on go-getter is going to address it.

[gmaghera]

True but by that token, maybe Terragrunt should be looking for a replacement.

[gmaghera]

I've tried to find an example in Terraform core where the go-getter issue could be reproduced, per @lorengordon's suggestion.

Here's the rough example, a main.tf file with this content:

module "gitlab_nested_group_example" {
  source = "git::https://gitlab.com/foo/bar/fred/terraform-modules/null.git//?ref=0.1.5"
  desired_count = 1
}

That works just fine, even though we know go-getter cannot handle such paths. The Terraform code base must be using something else to pull down modules hosted on gitlab.com. denis256 would it be possible to use the same path with Terragrunt?

Is there perhaps a better example I can try to reproduce? I cannot think off the bet of something analogous to terragrunt scaffold in signature, where a module URL is passed as a command argument.

[gmaghera]

@lorengordon apparently, someone did find a manifestation of the issue in Terraform! I'm hoping this results in traction.

See https://github.com/hashicorp/terraform/issues/34771#issuecomment-2270093251

[gmaghera]

Hello again.

Gruntwork, I would like to kindly ask that you consider pursuing a direct solution to this. Relying on getting a fix via Hashicorp's go-getter does not seem like a viable option. We have created an issue there, detailed the line of code where the issue stems from, placed a PR with the fix, yet there is no traction after months of waiting.

Scaffold is such a powerful feature! It bridges a gap in making Terragrunt accessible to engineers who may not have much experience with it, and it is also great for encouraging best practices. In its current form, however, the feature is broken for all GitLab enterprise customers.

CC: @josh-padnick @denis256 @yhakbar

[josh-padnick]

In its current form, however, the feature is broken for all GitLab enterprise customers.

@yhakbar @denis256 @levkohimins This alone makes it worth considering a hacky workaround until https://github.com/hashicorp/go-getter/issues/479 is merged (which could well be never).

But in addition, @gmaghera has really gone above and beyond to track down (and even attempt to resolve) this problem here. Can we address his underlying issue so he can begin using terragrunt scaffold with GitLab Enterprise?

[denis256]

I can take a look to add handling of Gitlab cases

[yhakbar]

Before you dig into this, @denis256 , could I get confirmation on a couple things to verify that we do, indeed need to workaround go-getter, @gmaghera ?

I know we already asked about this, and you already provided a response, so I'm sorry that I didn't ask clarifying questions about this early on. I didn't have a GitLab account, so I didn't check, but I just made one to confirm that SSH URLs don't work for nested paths:

$ go-getter '[email protected]:gitlab-org/ai-powered/eli5.git//eli5' eli5
2024/09/03 13:33:21 success!

It seems based on that example that go-getter v2 does in fact work, and we might just need to update the version used in the catalog command to a version that supports your use-case.

Do you mind running the following:

go install github.com/hashicorp/go-getter/cmd/go-getter@latest

Then trying out the standalone go-getter binary to see if there's still a need to workaround the usage of go-getter?

If all we need to do is upgrade the version of go-getter we're using, that will be a better solution than introducing a workaround for GitLab nested projects.

[gmaghera]

@yhakbar I will check if it works today.

But, looking at your test, that does not appear to be a nested repo, or not nested enough. You need at least three "layers" before the //.

For example gitlab.com/group1/group2/group3/repo.git//foobar. Go-getter wrongly assesses that group2 is the repository, or at least it used to.

I'll reply with my test results next.

[gmaghera]

@yhakbar can you please share how you've installed v2 of go-getter? Running go install github.com/hashicorp/go-getter/cmd/go-getter@latest installs v1.7.6. I've played with various @references to get it to install v2, but every attempt so far had failed.

[yhakbar]

Sorry, I may have been mistaken that I was using v2.

To be extra super confident that we're trying out v2 appropriately, please do the following:

$ git clone [email protected]:hashicorp/go-getter.git
$ cd go-getter
$ git checkout v2
$ cd cmd/go-getter
$ go install .

In my testing on an open source repo I was able to find that was more deeply nested, it continued to work:

$ go-getter [email protected]:gitlab-org/ai-powered/custom-models/pocs/gitlab-docs-indexer-poc.git poc
2024/09/04 14:02:06 -> poc

It's been a while since I used GitLab, so please let me know if I'm still missing a nuance here on the level of nesting for repos.