GitHub private repo archive URL now requires token in the header during Build

[jbrown-heroku]

Recently GitHub changed their access policy that you are no longer able to use URL query params for access_token to access private repos and instead must use HTTP header. This can cause issues with build functionality on Terraform Heroku Provider when source URL is a private GitHub archive link.

Terraform/Heroku Provider vers:

Terraform v1.0.7
on darwin_amd64
+ provider registry.terraform.io/heroku/heroku v4.6.0

Affected Resource(s)

• heroku_build (will also log an issue on Platform API as well)

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Use URL query params in source_blob.url when sourcing from a private GitHub repo

Important Factoids

Are there anything atypical about your accounts that we should know? For example: Running in EC2 Classic? Custom version of OpenStack? Tight ACLs? Using GitHub private repo, access via access_token

References

Breaking change took place (github) 9/8/21: https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/#changes-to-make

[mars]

Due to the dependency on a Heroku API (& Builds service) change for new authorization path with GitHub API, I strongly doubt this will be fixed in the elegant way wished for here.

The source.url attribute is designed only for public URLs:

Useful for building public, open-source source code, such as projects that publish releases on GitHub.

Embedding a secret key in the URL was never advised, as this is typically considered a bad security practice.

Alternative solution

Clone the private source repo before the Terraform run, and use heroku_build's source.path instead of source.url configuration, to point at the source repo's local directory.

• Script git clone of the private repo using Deploy Keys to support git ssh access to the private repo
• Follow the documented guidance for using source.path with a directory.

[mars]

✅ Using a script like this to populate the source.path before Terraform runs is the only way to solve this. More than a workaround, this is the solution. Terraform Cloud offers a pre-plan hook for this kind of solution.

---

Sample private repo clone script.

Expects env vars:

• GITHUB_DEPLOY_KEY, the private part of the deploy key
• GITHUB_SOURCE_REPO, the Github repo, example org-name/repo-name
• GITHUB_SOURCE_BRANCH, the branch to clone
• GITHUB_SOURCE_DIRECTORY, the directory in which to clone the repo.

#!/bin/bash
set -eu
set -o pipefail

echo "🔐  setup GitHub deploy key" >&2
mkdir -p ~/.ssh/
# Fix for "The authenticity of host 'github.com (…)' can't be established."
ssh-keyscan github.com >> ~/.ssh/known_hosts
# Save user's config value to ssh key file, named as default key so ssh will use it.
echo "$GITHUB_DEPLOY_KEY" > ~/.ssh/id_ed25519

echo "⬇️  clone private GitHub repo" >&2
git clone "[email protected]:${GITHUB_SOURCE_REPO}.git" --branch "$GITHUB_SOURCE_BRANCH" "$GITHUB_SOURCE_DIRECTORY"

Beware that ~/.ssh/id_ed25519 is the default ssh identity on my system. This may be different (such as id_rsa) for your target system.

[andoneve]

This workaround is great, thanks.

---

⚠️ Notice from maintainer: null_resource does not solve. Plan will always error because the source is not yet in-place.

---

I used it and ran into this issue when running terraform plan. I'm currently looking for a terraform equivalent to ansible's ignore_errors. Any help appreciated.

Error: Error stating build source path /tmp/source_code: stat /tmp/source_code: no such file or directory

Configuration:

locals {
  source_code = "/tmp/source_code"
}

resource "null_resource" "source_code" {
  provisioner "local-exec" {
    command = "mkdir -p ${local.source_code} && git clone '[email protected]:org/project.git' --branch main ${local.source_code}"
  }
}

resource "heroku_build" "initial_build" {
  app = heroku_app.app.id

  source {
    path = local.source_code
  }

  depends_on = [null_resource.source_code]
}

[mars]

@laurawadden that error indicates that /tmp/source_code does not exist, so you need to fix that to proceed without error.

[levivm]

@mars The issue, in this case, is when resource "heroku_build" "initial_build" { runs on the plan, it makes a stat command over the local.source_code path but it doesn't exist yet, so, it throws the error. I'm facing the same issue, any ideas? Even using the depends_on doesn't work.

@laurawadden did u find a solution for the issue?

[mars]

@levivm Clone the source before running Terraform.

[levivm]

@mars I tried to upload my code base to S3 and created a pre-signed URL with expiration and pass it to source.url. But the pre-signed URL always changes and it will download everything again. Even when there is no change, terraform will detect that there is a difference and run the build process again.

[levivm]

@mars So, If I want to build an heroku app from terraform and avoid downloading the code before, it needs to be a public file, it can't be from a private repo, right?

[mars]

@levivm then, perform that download before running Terraform, so that the source code is already at a consistent source.path.

[mars]

The options here are:

source.url, a public URL.

source.path, can come from anywhere, anyhow, but it needs to be in-place before running Terraform.

If either source.url or source.path change, then the resource is tainted and must be recreated/rebuilt.

[mars]

null_resource tricks might work to run a provisioner to fetch the source ahead of time, but be aware that null_resource has some really stupid behavior. Once null_resource is created, it will never run again, unless manually tainted or deleted from state. So, subsequent runs may be missing what its provisioner did, if Terraform is running on ephemeral compute.

[davidji99]

is created, it will never run again, unless manually tainted or deleted from state.

or you make use of the triggers attribute on null_resource.

[DanielViglione]

This does not work at all with null_resource. if you pass path a location that does not exist yet, it will error and so no file found. If you create an empty directory, pass it to "path", and then populate the contents of said directory with null_resource, then you will get this error:

Error: Provider produced inconsistent final plan

When expanding the plan for module.primary_app.heroku_build.build to include
new values learned so far during apply, provider
"registry.terraform.io/heroku/heroku" produced an invalid new value for
.local_checksum: was
cty.StringVal("SHA256:8a8f60ecb09b7e64c6d5214a8043865e608507db8c3f61f995eae6d078875901"),
but now
cty.StringVal("SHA256:9acc334f3554fac41c1f582d438cc5228dc89a07946594ee953fb5f74a548dd1").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Of course, I am not going to have my source code in a public repo. And it is highly inefficient to have to create an entirely different process outside of terraform to download it locally. If you use terragrunt, this won't even work at all.

[mars]

When this heroku_build / source.path feature was implemented, the expectation was that the source code for heroku_build is included along with the Terraform configuration, as subdirectories of the Terraform configuration (a monorepo).

The challenge with local source.path is determining when that source code has changed, so that build can be skipped if there are no changes to the source. A year ago, we made a significant change to that checksum algorithm to avoid build churn in ephemeral runtimes like Terraform Cloud and Heroku itself.

It's possible to make changes to the provider that would allow populating the source.path during the Terraform run:

• implement a new source attribute checksum_at_apply = true
• and the during apply, perform the checksum to determine whether to perform the build.

But, such a change would mean that heroku_build would always be tainted in the plan. Terraform would always see changes to apply.

The source.url approach does not suffer this problem, because the diff is based on the URL. If the URL changes, then the build is tainted.

The notion of supporting GitHub Deploy Keys (git+ssh) for private access to remote source seems good, but the standard way of doing that requires git CLI with ~/.ssh key setup on the local filesystem. Not friendly with generic Terraform usage across platforms. Maybe someone could implement this in Go (like this), but we would end up with the same dirty-plan problem as before, that the source.path diff would need to be deferred until apply, forever tainting the Terraform configuration with heroku_build changes.

So, @DanielViglione, the workaround to download via git+ssh script (or any other private access strategy) before Terraform runs is the solution here. In fact, thank you for your coarse comment that made me really reconsider the options and realize that the workaround is really the solution.

[dentarg]

While you would still have the tainted problem, the Download a repository archive (tar) GitHub API endpoint gives you (you'll have to extract it from the location header) a public URL (that expire after five minutes). That could be a bit more convenient than having to clone repos.

[mars]

✅ updated the solution to mention that,

Terraform Cloud now offers a pre-plan hook, the perfect place to run the source checkout, before the Terraform run.