Allow private provider registries to self-host provider packages/checksums/signatures under the same credentials as the registry itself

[BarakHc]

I have a remote Terraform registry that experiencing 401 error on the provider download (download_url, shasums_url and shasums_signature_url). It appears that only for those 3 downloads, the client does not add the authorization header.

I have one domain for the registry API, the modules and the providers files. All the other registry API calls are executed with the relevant domain authorization token, as expected. The modules downloads are executed with the creds in the .netrc file, as expected. But the download of the SHA256SUMS, .sig and provider.zip files are executed with no authorization header at all.

The host of the registry and the download links are the same.
(The token for the domain is updated in the credentials.tfrc.json and the creds are in the .netrc file)

I am working with a reverse-proxy between the client to the registry so I have a record of all the request&responses with their headers.

main.tf file:

terraform {
 required_providers {
   aws = {
     source = "terraform-remote.mydomain.com/hashicorp/aws"
   }
 }
}
 
module "vpc" {
 source  = "terraform-remote.mydomain.com/terraform-aws-modules/vpc/aws"
 version = "2.21.0"
}

ClientOutTrace.txt

Reverse proxy record + the v1/providers/hashicorp/aws/3.53.0/download/darwin/amd64 response:

Registry call headers (200)

The 401 download request headers:

Terraform version: 1.0.4

[apparentlymart]

Hi @BarakHc! Thanks for reporting this.

I guess this is the provider registry equivalent of #28659, where we were discussing the same behavior for module installation. Again, unfortunately this behavior is intended rather than a bug because from Terraform's perspective the registry is just a pointer to the location for the files, and the files themselves are treated as an entirely separate system.

As in that other issue, I do see the benefit of having a way to treat the package files as if they are "part of the registry" with the same credentials, and I think it would make sense for us to try to solve both of these in the same way so that the opt-in protocol is consistent across both registry types.

However, I'm going to leave both issues open because they do represent to distinct use-cases requiring separate implementation to address, even if the two implementations have some protocol features in common. One difference for provider installation compared to direct-HTTP-based module installation is that, IIRC, provider installation doesn't honor .netrc, and so perhaps there's also an interim solution here of adding NetRC support to the provider installer so that the explicit separate configuration you tried would work, although in the long run I'd still like to have a means for a private provider registry to host packages itself (on its own hostname) without any special configuration on the end-user's part.

Since we're discussing a new feature which needs to be designed I'm going to relabel this as an enhancement just to send it over into our feature request process where we consider changes to Terraform's design, as opposed to Terraform's implementation of the existing design.

As with the other issue, in the meantime the approach for current versions of Terraform will be the same as I suggested in the other issue: make your provider registry generate a URL containing a time-limited, user-specific signature (or some other suitable credential) and then have the server which handles that request verify the credential before returning the package.

Thanks again for sharing this!

[BarakHc]

Hi @apparentlymart , Thank you for the informative answer!

[k0da]

Nice I'm not alone with this use case. My use case is similar but we use github releases as storage backend. Once provider is built GH action triggers Provider registry generator which publishes provider metadata into github pages.

So we're fine to have registry token the same as download token. So far we're on the same page here. I think this could be a good point to start with. Later once another usecase appears with requirement to have different token for download, it could be extended with lets say dowloadToken within host credential structure.

I have implemented and successfully tested my case within https://github.com/hashicorp/terraform/pull/29642 I'd love to see any feedback on taken approach.

[k0da]

Any feedback?

[aaklilu]

@apparentlymart here's the interim solution you suggested

[apparentlymart]

Hi @aaklilu! Thanks for working on that.

I'm not on the Terraform team anymore, so I'll need to leave it to someone still on the team to consider what you submitted.

(FWIW, It seems like https://github.com/hashicorp/terraform/issues/36011 is effectively a duplicate of this issue and https://github.com/hashicorp/terraform/issues/28659 but framed as a problem with the documentation.)

[liamcervante]

@aaklilu's .netrc workaround for this issue has been merged. I'm not closing this issue as I think the .netrc workaround is not a true fix for this and we might want to implement something properly in the future but I don't know what that will look like.

For the time being, from Terraform v1.11 going forward you'll be able to use a .netrc file to attach credentials to follow up provider requests should that be required.

[aaklilu]

@aaklilu's .netrc workaround for this issue has been merged. I'm not closing this issue as I think the .netrc workaround is not a true fix for this and we might want to implement something properly in the future but I don't know what that will look like.

For the time being, from Terraform v1.11 going forward you'll be able to use a .netrc file to attach credentials to follow up provider requests should that be required.

Thanks @liamcervante, I believe this is an acceptable workaround for us now...at least there's a way.

[apparentlymart]

FWIW what I had in mind when writing my earlier comments was to extend the Find a Provider Package response format with a new optional property like "use_registry_host_credentials" which can be set to true if the registry trusts the URLs given in "download_url", "shasums_url", and "shasums_signature_url" enough to disclose the user's registry host auth token to them.

If that were set then Terraform could arrange to send the same Authorization: Bearer ... header field it included in the registry API request when fetching the indicated assets. If it were not set then the current behavior would be preserved, and thus Terraform would not begin disclosing the registry host's bearer token to anywhere it wasn't previously disclosed.

---

I don't know if the relevant parts of the Terraform code have changed significantly since I made that comment, but FWIW with things how they used to be I'd change getproviders.PackageHTTPURL to be a struct with separate fields for URL and optional HTTP credentials information, and then to change getproviders.registryClient.PackageMeta to notice that new property and populate the credentials field of getproviders.PackageHTTPURL accordingly if it's set.

getproviders.registryClient.PackageMeta would also need to add the credentials to the requests it makes to fetch the SHA256SUMS file and its associated signature.

I offer this idea just in case it's helpful to understand what I was getting at before. The Terraform team might have a different preference on whether and how to solve this, so I would recommend against anyone trying to write a PR based on what I've described unless someone on the Terraform team suggests that it would be welcome.

(The equivalent of this for modules is probably harder to achieve because of the involvement of go-getter and the weird way the module registry protocol reports the source address using a custom HTTP response header field. I'll leave that one as an exercise for the reader. :stuck_out_tongue_winking_eye:)

[fancybear-dev]

@apparentlymart it seems that getterHTTPGetter (in go-getter) used to retrieve modules in getmodules, has support already for NetRC credentials. It tries to add credentials for Get and GetFile methods using addAuthFromNetrc. I haven't tested it myself, but there might be chance that with these recent changes for providers, it's also possible to obtain modules from a private registry using NetRC (HTTP basic auth).

We were having issues with an identity proxy that uses oauth, so NetRC won't be usable for us (given it's HTTP Basic Auth). Hopefully at some point dynamic Authorization headers will be implemented for private registry use.

I have tested the following private patches for the Terraform CLI for the provider methods that simply does the TF_TOKEN (https://developer.hashicorp.com/terraform/cli/config/config-file#environment-variable-credentials) check and adds headers at the right places which works, but doubt Hashicorp would accept them as fixes (do share if they would be acceptable) - as they are symptomatic fixes. If you want a proper fix, you'd need to refactor how clients are used throughout the codebase imo.

If you want to extend this to modules, a similar solution would need to be implemented in the go-getter getterHTTPGetter method, which can be done with as little as like 10 lines of additional code. Comes down again to if it would be acceptable to Hashicorp or not. Perhaps I'll create a PR for it later down the line.

The (dirty) patches required to implement TF_TOKEN for providers;

diff --git a/internal/getproviders/registry_client.go b/internal/getproviders/registry_client.go
index 82ab2ed873..1072a7b047 100644
--- a/internal/getproviders/registry_client.go
+++ b/internal/getproviders/registry_client.go
@@ -433,7 +433,13 @@ func (c *registryClient) errUnauthorized(hostname svchost.Hostname) error {
 }

 func (c *registryClient) getFile(url *url.URL) ([]byte, error) {
-       resp, err := c.httpClient.Get(url.String())
+       req, err := retryablehttp.NewRequestWithContext(context.Background(), "GET", url.String(), nil)
+       if err != nil {
+               return nil, err
+       }
+       // Inject headers including the Authorization header from TF_TOKEN_<host>
+       c.addHeadersToRequest(req.Request)
+       resp, err := c.httpClient.Do(req)
        if err != nil {
                return nil, err
        }

diff --git a/internal/getproviders/registry_client.go b/internal/getproviders/registry_client.go
index 82ab2ed873..95dfd42a55 100644
--- a/internal/getproviders/registry_client.go
+++ b/internal/getproviders/registry_client.go
@@ -18,6 +18,7 @@ import (
        "path"
        "strconv"
        "time"
+        "strings"

        "github.com/hashicorp/go-retryablehttp"
        svchost "github.com/hashicorp/terraform-svchost"
@@ -407,10 +408,19 @@ FindMatch:
 }

 func (c *registryClient) addHeadersToRequest(req *http.Request) {
-       if c.creds != nil {
-               c.creds.PrepareRequest(req)
-       }
-       req.Header.Set(terraformVersionHeader, version.String())
+       // Prepare credentials if they exist.
+       if c.creds != nil {
+           c.creds.PrepareRequest(req)
+       }
+       // Set the Terraform version header.
+       req.Header.Set(terraformVersionHeader, version.String())
+
+       // Construct a host-specific environment variable name.
+       // For example, for a host "registry.terraform.io", the env var is TF_TOKEN_registry_terraform_io.
+       tokenEnvVar := "TF_TOKEN_" + strings.ReplaceAll(c.baseURL.Host, ".", "_")
+       if token := os.Getenv(tokenEnvVar); token != "" {
+           req.Header.Set("Authorization", "Bearer " + token)
+       }
 }

 func (c *registryClient) errQueryFailed(provider addrs.Provider, err error) error {

diff --git a/internal/providercache/package_install.go b/internal/providercache/package_install.go
index 762b149adb..6aafa5529c 100644
--- a/internal/providercache/package_install.go
+++ b/internal/providercache/package_install.go
@@ -7,8 +7,10 @@ import (
        "context"
        "fmt"
        "net/url"
+        "net/http"
        "os"
        "path/filepath"
+        "strings"

        getter "github.com/hashicorp/go-getter"

@@ -26,17 +28,7 @@ var unzip = getter.ZipDecompressor{}

 func installFromHTTPURL(ctx context.Context, meta getproviders.PackageMeta, targetDir string, allowedHashes []getproviders.Hash) (*getproviders.PackageAuthenticationResult, error) {
        urlStr := meta.Location.String()
-
-       // When we're installing from an HTTP URL we expect the URL to refer to
-       // a zip file. We'll fetch that into a temporary file here and then
-       // delegate to installFromLocalArchive below to actually extract it.
-       httpGetter := getter.HttpGetter{
-               Client:                httpclient.New(),
-               Netrc:                 true,
-               XTerraformGetDisabled: true,
-       }
-
-       urlObj, err := url.Parse(urlStr)
+        urlObj, err := url.Parse(urlStr)
        if err != nil {
                // We don't expect to get non-HTTP locations here because we're
                // using the registry source, so this seems like a bug in the
@@ -50,7 +42,25 @@ func installFromHTTPURL(ctx context.Context, meta getproviders.PackageMeta, targ
        defer f.Close()
        defer os.Remove(f.Name())

-       archiveFilename := f.Name()
+        // When we're installing from an HTTP URL we expect the URL to refer to
+        // a zip file. We'll fetch that into a temporary file here and then
+        // delegate to installFromLocalArchive below to actually extract it.
+        httpGetter := getter.HttpGetter{
+                Client:                httpclient.New(),
+                Netrc:                 true,
+                XTerraformGetDisabled: true,
+        }
+
+       host := urlObj.Host
+       tokenEnvVar := "TF_TOKEN_" + strings.ReplaceAll(host, ".", "_")
+       if token := os.Getenv(tokenEnvVar); token != "" {
+               if httpGetter.Header == nil {
+                       httpGetter.Header = make(http.Header)
+               }
+               httpGetter.Header.Set("Authorization", "Bearer "+token)
+       }
+
+        archiveFilename := f.Name()
        err = httpGetter.GetFile(archiveFilename, urlObj)
        if err != nil {
                if ctx.Err() == context.Canceled {

[oefimov]

I think the interim solution (enabling .netrc) currently is incomplete for providers:

• download url - DONE - thanks to @aaklilu 35843
• shasums url - TODO - registry_client.go
• shasums signature url - TODO - same as above

@liamcervante @aaklilu can someone help with this plz? I'm happy to contribute, just not quite familiar with this codebase yet.