question: how to handle dependent terragrunt.hcl files in other repositories

[dgokcin]

Hello and thank you for this project.

We are at a phase where we are dividing our apps into microservices which will have their own repositories. We were using terragrunt with 2 aws accounts production and qa in a single repository called Infrastructure and all of our infra is defined there. In our infrastructure we have some common modules like the vpc, alb etc that other resources are dependent to. When things are in the same repository, this dependency is not a problem but once we divide the infrastructure, we will have a single repo for storing the terraform modules and terragrunt files for the common resources, and and many more app repositories which contain source code and resources needed for that app.

To solve this problem locally, the first solution that came up to my mind was making sure that both the common infra repo and app repo is cloned to my machine and at the same directory and referencing to the common directories with the dependency block and relative paths. I can do that easily on my machine but it feels a little bit hacky. What should I do If I have a single vpc and two apps that need the id of that vpc(or any other output of any other common resource)

[suppix]

Just an idea, maybe you can try to have common code as a submodule in git, in this case all apps can use "shared" code as a submodule.

[yorinasub17]

We unfortunately don't have good solutions to offer here at the moment, but are actively thinking about ways to do cross remote project links in terragrunt, including how to handle remote includes. But that will take some time before we implement it, so in the meantime, your best option is unfortunately to have the clones locally, or use other mechanisms like data source lookups to link the two projects.

[dgokcin]

@yorinasub17 thank you for the reply.

I decided to use submodules to work around this problem just like @suppix suggested. Each repository has its infrastructure defined inside the repo and the common infra repository is added as a submodule to each repo.

It would be great if I could use a remote path instead of a relative path for the dependency block, just like the terraform blocks.

[mauxas]

Hey guys maybe its planned to fix this soon?

[robcortez]

Just checking in on this one and wondering if it's on the roadmap.

This functionality would be a very useful tool to help keep things DRY.

[vadanx]

Yes, this would be amazing for our use cases as well.

[g-getsov]

This is the holy grail for me right now.

[djha-skin]

WORKAROUND:

1. Put this in your .gitignore for the depending terragrunt module:

# Remote modules
/remote_modules

2. Add this script to the root of your terragrunt module under the namne fetch_terragrunt_module.sh:

#!/bin/sh

set -e

usage() {
    echo "Usage: $0 <checkout_directory> <directory> <repo> <ref> <subdir>" >&2
    exit
}
if [ -z "${1}" ]
then
    usage
fi

checkout_directory="${1}"
shift
directory="${1}"
shift
repo="${1}"
shift
ref="${1}"
shift
subdir="${1}"
shift

if [ ! -d "${checkout_directory}" ]
then
  mkdir -p "${checkout_directory}"
fi

cd "${checkout_directory}"
root_path="${PWD}"

if [ ! -d "${directory}" ]
then
    git clone "${repo}" "${directory}" >/dev/null 2>&1
    cd "${directory}"
    git checkout "${ref}" >/dev/null 2>&1
    cd "${root_path}"
fi

printf '%s' "${checkout_directory}/${directory}/${subdir}"

3. Use the run_cmd() function to check out the stuff and give the name of the directory:

dependency "legacy_remote_state" {
    config_path = run_cmd("./fetch_terragrunt_module.sh",
                          "remote_modules",
                          "my-git-repo-name",
                          "[email protected]:My-Organization/my-remote-terragrunt-stuff.git",
                          "git-ref",
                          "subdir-within-the-git-repo")
}

[dgokcin]

update: to workaround my original problem, I first used submodules for a while but as the repository count improved in our organization, it became a problem to manually update the submodules so we introduced dependabot for automatically updating the submodules with PRs in a controlled manner. This also worked fine for a while but after a while I found myself approving and merging a lot of dependabot prs. As the final improvement, we have used a custom github actions workflow that automatically approves and merges dependabot prs for submodules only and that worked great!

[digorgonzola]

Hey @dgokcin ! We are facing the same challenge. I'd be keen to know more about how you use dependabot to update the submodules. Are you able to share some sample code from your workflow? I'm not a huge of git submodules so the remote "include" functionality in Terragrunt would be a really nice improvement.

[dgokcin]

Hello @digorgonzola! I worked around this limitation by first adding my common Infrastructure repository to my app repository as a submodule. Afterwards, I used a Dependabot config similar to the example below.

version: 2
updates:
  - package-ecosystem: "gitsubmodule"
    open-pull-requests-limit: 5
    directory: "/"
    target-branch: main
    schedule:
      interval: "daily"
    reviewers:
      - "OrganizationName/TeamName"

This allowed me to have automatic updates on submodules. It worked okay for a while, but as the number of repositories in our organization increased, updating the Infrastructure repository meant manually approving and merging dozens of Dependabot PRs, which was really annoying.

So, I came up with a basic, reusable GitHub Actions workflow that automatically approves and merges Dependabot PRs, but only for submodules.

Unfortunately, I cannot share the full workflow, but I think, given the actions and how I used them, you can figure it out.

• This conditional at the job level is used to ensure that the action is only triggered for Dependabot PRs.

    if: |
      github.actor == 'dependabot[bot]' &&
      !github.event.pull_request.draft

• dependabot/fetch-metadata@v1: This step is used to extract the package ecosystem, submodules in our case, from the Dependabot PR.
• hmarr/auto-approve-action@v3: Used to auto-approve a PR if contains(inputs.package-ecosystem, steps.dependabot-metadata.outputs.package-ecosystem).
• Merging the PR with GitHub CLI, admin mode:

      - name: Force-merge for Dependabot PRs
        if: contains(inputs.package-ecosystem, steps.dependabot-metadata.outputs.package-ecosystem)
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GITHUB_TOKEN: ${{ steps.get_workflow_token.outputs.token }}
        run: gh pr merge --admin --merge "$PR_URL"

Hope that it helps!