Testing Terragrunt

[daanbosch]

I keep getting the following error:

Error locking project: ResourceNotFoundException: Requested resource not found

Any way to debug what is going on?

projects:
- name: 00_myvm
  dir: dev/env/00_myvm
  terragrunt: true

[ZIJ]

hey @daanbosch thanks for filing the issue! did you add your cloud provider keys as secrets in the action? (assuming you are using it with Github Actions)

also - what other log outputs are there?

[daanbosch]

Full logs:

Run diggerhq/tfrun@main
Run cd /runner/_work/_actions/diggerhq/tfrun/main
go: downloading github.com/aws/aws-sdk-go v1.44.219
go: downloading cloud.google.com/go/storage v1.30.1
go: downloading github.com/google/go-github/v50 v50.1.0
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/hashicorp/terraform-exec v0.18.1
go: downloading github.com/hashicorp/terraform-json v0.15.0
go: downloading github.com/hashicorp/go-version v1.6.0
go: downloading cloud.google.com/go v0.110.0
go: downloading golang.org/x/oauth2 v0.6.0
go: downloading golang.org/x/crypto v0.5.0
go: downloading github.com/google/go-querystring v1.1.0
go: downloading github.com/zclconf/go-cty v1.13.0
go: downloading cloud.google.com/go/iam v0.12.0
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/googleapis/gax-go/v2 v2.7.1
go: downloading google.golang.org/api v0.114.0
go: downloading cloud.google.com/go/compute/metadata v0.2.3
go: downloading cloud.google.com/go/compute v1.18.0
go: downloading google.golang.org/genproto v0.0.0-20230320184635-7606e756e683
go: downloading google.golang.org/grpc v1.53.0
go: downloading google.golang.org/protobuf v1.29.1
go: downloading go.opencensus.io v0.24.0
go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
go: downloading golang.org/x/net v0.8.0
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading github.com/apparentlymart/go-textseg/v13 v13.0.0
go: downloading golang.org/x/text v0.8.0
go: downloading github.com/golang/protobuf v1.5.2
go: downloading golang.org/x/sys v0.6.0
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.3
go: downloading github.com/google/go-cmp v0.5.9
go: downloading github.com/jmespath/go-jmespath v0.4.0
GitHub context parsed successfully
Digger config read successfully
Lock provider has been created successfully
GitHub event processed successfully
GitHub event converted to commands successfully
Lock team-<redacted>/<redacted>#00_myvm
2023/04/06 10:57:43 Error locking project: ResourceNotFoundException: Requested resource not found
Error: Process completed with exit code 1.

[daanbosch]

did you add your cloud provider keys as secrets in the action?

Yes using cloud creds of a supported provider and I share them as env vars.

[ZIJ]

Looking into it! Also created an issue to improve debuggability (above)

So far it seems to be coming from here: https://github.com/diggerhq/digger/blob/46a2a6d8dc45591c73885943a3faa97723f3cf43/pkg/digger/digger.go#L233

from there its either GCP locking provider or AWS locking provider that's throwing this error. We should capture this as debug info but for the time being need to resort to asking - which cloud provider are you using?

[ZIJ]

Ok we were able to isolate it to AWS locks implementation as the most likely source. It suggests that the DynamoDB table for locks likely wasn't created:

https://github.com/diggerhq/digger/blob/46a2a6d8dc45591c73885943a3faa97723f3cf43/pkg/aws/dynamo_locking.go#L50

One possible reason is the AWS keys not having enough permissions to do that.

@daanbosch What is the scope of permissions for your AWS keys?

[daanbosch]

It has full access to the db, my guess it is trying to lock something that does not exist.

[ZIJ]

Okay then it's likely something wrong with the DynamoDB expression building here: https://github.com/diggerhq/digger/blob/46a2a6d8dc45591c73885943a3faa97723f3cf43/pkg/aws/dynamo_locking.go#L43

We will obviously refactor this (#113) - but for the time being, can you share the entries that you have in your DynamoDB table currently?

[ZIJ]

If it contains anything that you are not comfortable sharing publicly - here's our Slack

[daanbosch]

I can't join your slack, it's limited to certain email addresses. What I did find was the following in Dynamodb:

"<redacted>/<redacted>/<redacted>/<redacted>/00_myvm/terraform-bm.tfstate-md5"

Could it be that it can't detect the custom name of the state? terraform-bm.tfstate the -bm part I add with Terragrunt?

Edit: No, that's not it I think, I reverted this change already, this is on old entry.

[daanbosch]

Maybe it could be related to me using this Terragrunt fucntion for the key path:

${path_relative_to_include()}

Docs: https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#path_relative_to_include

[motatoes]

Hey @daanbosch I think there is a bug in dynamoDB creation I'm looking into it now and. I will keep you updated soon

[motatoes]

Hey @daanbosch I tested it out in this demo repository https://github.com/diggerhq/digger_demo_dynamotest/pull/1 and indeed we are not creating the repo on first run. I reproduced in this run by using the main action: https://github.com/diggerhq/digger_demo_dynamotest/actions/runs/4630926317/jobs/8193148451

After switching over to the branch action. Some more testing and improvement needs to be done but in the meantime you can test it by switching to the branch and try it out. It could still not work on the first run due to race condition but on second run it should work. You can just change the plan.yml in your main branch and then comment digger plan in your existing pull request to trigger the job.

In the meantime I'm getting more reviews and tests written on this PR ..

[daanbosch]

Okay, it worked the second time, I can see it being locked now. However it is complaining that it can't find Terraform. Should I install this myself on the runner? Should I install Terragrunt? I'm using an internal runner, because I need to access internal services.

time=2023-04-11T08:21:27Z level=error msg=exec: "terraform": executable file not found in $PATH
time=2023-04-11T08:21:27Z level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1
2023/04/11 08:21:27 Error executing plan: error: exit status 1

[ZIJ]

It is odd that it can't find Terraform - possibly a bug on private GH runners. Tagging @motatoes

Is Terragrunt installed in the workflow prior to Digger run?

Similar to this: https://github.com/diggerhq/digger_demo_terragrunt/blob/main/.github/workflows/plan.yml

[motatoes]

Hey @daanbosch I'm glad that it has worked, I will continue working on that pull request. We need to perform testing better on private runners. In the default github image terraform comes installed in the image. We also have it in the action to install terragrunt prior to digger: https://github.com/diggerhq/digger_demo_terragrunt/blob/main/.github/workflows/plan.yml#L38

I am not sure if this works well in private runners though. Would you be willing to help us test out digger in private runners? If you are then I invite you to join our slack and we can get more feedback there regarding your setup: Chat on Slack

[daanbosch]

@ZIJ I have the following:

      - uses: actions/setup-go@v4
        with:
          go-version: '>=1.20.3'

      - name: Setup Terragrunt
        uses: autero1/[email protected]
        with:
          terragrunt_version: 0.45.0
          token: <redacted>

So yes, it should be installed I think? I'm using a Ubuntu based runner.

[daanbosch]

Okay, i added the following steps and now it works:

      - name: Create directory for digger
        run: |
          mkdir -p /usr/local/bin/
          sudo chown runner: /usr/local/bin/
          sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
          wget -O- https://apt.releases.hashicorp.com/gpg | \
          gpg --dearmor | \
          sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
          echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
          https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
          sudo tee /etc/apt/sources.list.d/hashicorp.list
          sudo apt update
          sudo apt-get install terraform



      - uses: actions/setup-go@v4
        with:
          go-version: '>=1.20.3'

      - name: Setup Terragrunt
        uses: autero1/[email protected]
        id: setup_terragrunt
        with:
          terragrunt_version: 0.45.0
          token: ${{ secrets.GITHUB_TOKEN }}  

      - name: Add Terrragrunt to path
        run: |
          echo "Adding the following to runner PATH" ${{ steps.setup_terragrunt.outputs.terragrunt_path }}
          echo ${{ steps.setup_terragrunt.outputs.terragrunt_path }} >> $GITHUB_PATH
          

      - name: digger tfrun
        uses: diggerhq/tfrun@feat/dynamo-table-creation
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}    
          terragrunt: true
          DEBUG: true

[daanbosch]

Now I run into my next problem, I'm trying to download a module from a different private repo. What would be the best approach to give access to the action?

error downloading 'ssh://[email protected]/my-org/my-module.git?ref=v0.0.2'

[motatoes]

hey @daanbosch That's great that you got this far setting it up in the private runner. For private modules hosted on github my intiuition is that we need a way to get the ssh keys into your private runner. I'm going to do some more digging 🚚 and find out the options we have to set this up

[daanbosch]

Added a deploy_key to the ssh agent, this works for me, although some "batteries included" action from digger would be helpful for this.

[daanbosch]

In the end I was not able to get it to work because of: https://github.com/diggerhq/digger/issues/129

@motatoes how where you able to apply in your tests with Terragrunt if it asks for input?

[motatoes]

@daanbosch I just checked the test, it seemed that plan worked but apply did not work but I didn't notice that it was stalled on entering a value 😅

[Spartakovic]

@daanbosch to access private github modules we can pack this action (https://github.com/webfactory/ssh-agent#support-for-github-deploy-keys) as part of our action opaquely. You will still need to provide a deploy key, but no other setup will be required probably. Will this work for you?

[daanbosch]

Yes, it is exactly what I'm using at the moment