WIP - feat(dependency): implement outputs fetching from gcs

[TPXP]

Description

This implements outputs fetching from the GCS backend, in a similar fashion to what has been implemented for S3.

I've also added some misc fixes:

• correct typo: steateBody -> stateBody
• drop unused Path parameter on GCS config (it's not supported)

Tested on my architecture, everything seems to work!

TODOs

Read the Gruntwork contribution guidelines.

• [x] Update the docs.
• [x] Run the relevant tests successfully, including pre-commit checks.
• [x] Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added support for the GCS backend for dependency output fetching (--terragrunt-fetch-dependency-output-from-state)

Migration Guide

None, this makes this error message go away (and speeds up dependency output state resolution)

ERRO[0001] FetchDependencyOutputFromState is not supported for backend gcs, falling back to normal method  prefix=[something]

[denis256]

Hello, looks like goimports should be executed:

[INFO] Initializing environment for https://github.com/gruntwork-io/pre-commit.
Terraform fmt............................................................Passed
goimports................................................................Failed
- hook id: goimports
- files were modified by this hook

config/dependency.go
config/dependency.go
config/dependency.go


Exited with code exit status 1

[denis256]

It would be helpful to add integration tests for these changes. This will allow us to track and verify that the functionality works end-to-end in the future.

[TPXP]

Hello, thanks a lot for the prompt feedback 🙌

I forgot to run pre-commit indeed, my bad. I ran it again, it looks better now.

Regarding testing, I'm not sure how to proceed. I couldn't find tests for getTerragruntOutputJsonFromRemoteStateS3. The easiest seems to be to use an integration test with dependencies?

[denis256]

INFO [runner] linters took 52.66096551s with stages: goanalysis_metalinter: 52.642627041s 
config/dependency.go:946:20: Error return value of `reader.Close` is not checked (errcheck)
        defer reader.Close()
                          ^

[sonarqubecloud[bot]]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[TPXP]

Hello, thanks for getting back to me 😃

I looked into the CI and its tests a little more and ran goimports, golint as well as a few tests. Hopefully all is good now 🤞

[yhakbar]

Hey @TPXP ,

It looks like this PR needs to be rebased off master. Please mark the PR as ready for review when it is.

[TPXP]

Hello, thanks for getting back to me. I've rebased the PR on the latest changes, let me know if there's anything else I can do to help :)

[denis256]

Hello, looks like PR still doesn't have any tests, it will be impossible to track if this functionality continues to work without automated tests

[TPXP]

Hey @denis256 , thanks for your reply.

I tried to adapt the AWS tests using this feature and added them to this PR. I can't say for sure if they work as I couldn't get them to run on my machine unfortunately (it seems the CI uses a magic run-go-tests program). Let me know how that looks 🙏

[denis256]

Looks like integration_gcp_test.go need formatting

pre-commit installed at .git/hooks/pre-commit
[INFO] Initializing environment for https://github.com/gruntwork-io/pre-commit.
Terraform fmt............................................................Passed
goimports................................................................Failed
- hook id: goimports
- files were modified by this hook

test/integration_gcp_test.go

[yhakbar]

This PR has gone stale, @TPXP . Would you be willing to resolve the merge conflicts, then address the formatting issue @denis256 raised?

[TPXP]

Hello @yhakbar, apologies for the delay. I took some time to address @denis256's feedback and fixed the merge conflicts.

• The code still works on my GCS setup
• It has a test (which I couldn't run as I don't have run-go-tests)
• pre-commit run --all and make run-lint pass

Feel free to edit my branch to speed up review 🙏

[TPXP]

Thanks for the testing @yhakbar , I've updated the code with your suggestion 👍

[denis256]

    integration_gcp_test.go:203: [terragrunt init --terragrunt-fetch-dependency-output-from-state --terragrunt-non-interactive --terragrunt-working-dir /tmp/terragrunt-test1920290761/fixtures/gcs-output-from-remote-state/env1/app2 --terragrunt-disable-log-formatting]
    integration_gcp_test.go:203: [TestGcpMockOutputsFromRemoteState] Full contents of stdout:
    integration_gcp_test.go:203: [TestGcpMockOutputsFromRemoteState] 
    integration_gcp_test.go:203: [TestGcpMockOutputsFromRemoteState] Full contents of stderr:
    integration_gcp_test.go:203: [TestGcpMockOutputsFromRemoteState] 
    integration_gcp_test.go:204: 
        	Error Trace:	/home/circleci/project/test/integration_gcp_test.go:204
        	Error:      	Received unexpected error:
        	            	storage: object doesn't exist
        	Test:       	TestGcpMockOutputsFromRemoteState
    panic.go:629: Deleting test GCS bucket terragrunt-test-bucket-5rrfxd
--- FAIL: TestGcpMockOutputsFromRemoteState (10.99s)

Looks like it still fails

[yhakbar]

Hey @TPXP ,

This PR has gone stale again, so I'm going to move it to draft, and mark it WIP.

When you've done the following, please mark the PR as ready for review again:

• Rebase off main
• Addressed @denis256 's comments
• Run lints locally (including pre-commit checks)
• Run whatever tests you can locally

[pnivlek]

Hi. Good work on this so far, looking forward to it. It looks like it's been (coincidentally) a year since this was marked stale, with a decent amount of changes to upstream. Are you still working on this @TPXP? If not, I'm happy to take over rebasing/updating tests for it.