hashicorp
Allow private module registries to self-host module packages under the same credentials as the registry itself
I try to use citizen as a private registry, with apache httpd to handle the Authorization header.
GET requests to /v1/modules/eric/rds-oracle/aws/versions and /v1/modules/eric/rds-oracle/aws/0.1.0/download contain the token from the terraform.rc file. The call to /v1/modules/tarball/eric/rds-oracle/aws/0.1.0/module.tar.gz does not contain a token and causes this ouput:
λ terraform init
Initializing modules...
Downloading citizen.some.domain/eric/rds-oracle/aws 0.1.0 for eric...
╷
│ Error: Failed to download module
│
│ Could not download module "eric" (main.tf:1) source code from "https://citizen.some.domain/v1/modules/tarball/eric/rds-oracle/aws/0.1.0/module.tar.gz": bad response code:
│ 401.
Terraform Version
Terraform v0.15.3
on windows_amd64
BTW: terraform 0.13.5 comes with the same error.
Terraform Configuration Files
module "eric" {
source = "citizen.some.domain/eric/rds-oracle/aws"
version = "0.1.0"
}
In the terraform.rc file:
credentials "citizen.some.domain" {
token = "someToken"
}
Debug Output
2021-05-09T19:18:56.870+0200 [DEBUG] Adding temp file log sink: C:\Users\eric\AppData\Local\Temp\terraform-log541667279
2021-05-09T19:18:56.871+0200 [INFO] Terraform version: 0.15.3
2021-05-09T19:18:56.871+0200 [INFO] Go runtime version: go1.16.2
2021-05-09T19:18:56.871+0200 [INFO] CLI args: []string{"c:\\ontwikkel\\bin\\terraform.exe", "init"}
2021-05-09T19:18:56.872+0200 [TRACE] Stdout is a terminal of width 176
2021-05-09T19:18:56.872+0200 [TRACE] Stderr is a terminal of width 176
2021-05-09T19:18:56.872+0200 [TRACE] Stdin is a terminal
2021-05-09T19:18:56.878+0200 [DEBUG] Attempting to open CLI config file: C:\Users\eric\AppData\Roaming\terraform.rc
2021-05-09T19:18:56.878+0200 [INFO] Loading CLI configuration from C:\Users\eric\AppData\Roaming\terraform.rc
2021-05-09T19:18:56.881+0200 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-05-09T19:18:56.881+0200 [DEBUG] ignoring non-existing provider search directory C:\Users\eric\AppData\Roaming\terraform.d\plugins
2021-05-09T19:18:56.883+0200 [DEBUG] ignoring non-existing provider search directory C:\Users\eric\AppData\Roaming\HashiCorp\Terraform\plugins
2021-05-09T19:18:56.883+0200 [INFO] CLI command args: []string{"init"}
2021-05-09T19:18:56.885+0200 [TRACE] ModuleInstaller: installing child modules for . into .terraform\modules
2021-05-09T19:18:56.885+0200 [DEBUG] Module installer: begin eric
2021-05-09T19:18:56.885+0200 [TRACE] ModuleInstaller: eric is not yet installed
2021-05-09T19:18:56.885+0200 [TRACE] ModuleInstaller: cleaning directory .terraform\modules\eric prior to install of eric
2021-05-09T19:18:56.885+0200 [TRACE] ModuleInstaller: eric is a registry module at citizen.some.domain/eric/rds-oracle/aws
2021-05-09T19:18:56.885+0200 [DEBUG] eric listing available versions of citizen.some.domain/eric/rds-oracle/aws at citizen.some.domain
2021-05-09T19:18:56.886+0200 [DEBUG] Service discovery for citizen.some.domain at https://citizen.some.domain/.well-known/terraform.json
2021-05-09T19:18:56.886+0200 [TRACE] HTTP client GET request to https://citizen.some.domain/.well-known/terraform.json
2021-05-09T19:18:57.076+0200 [DEBUG] fetching module versions from "https://citizen.some.domain/v1/modules/eric/rds-oracle/aws/versions"
2021-05-09T19:18:57.076+0200 [DEBUG] GET https://citizen.some.domain/v1/modules/eric/rds-oracle/aws/versions
2021-05-09T19:18:57.076+0200 [TRACE] HTTP client GET request to https://citizen.some.domain/v1/modules/eric/rds-oracle/aws/versions
2021-05-09T19:18:57.111+0200 [DEBUG] found available version "0.1.0" for eric/rds-oracle/aws
2021-05-09T19:18:57.112+0200 [DEBUG] looking up module location from "https://citizen.some.domain/v1/modules/eric/rds-oracle/aws/0.1.0/download"
2021-05-09T19:18:57.112+0200 [DEBUG] GET https://citizen.some.domain/v1/modules/eric/rds-oracle/aws/0.1.0/download
2021-05-09T19:18:57.112+0200 [TRACE] HTTP client GET request to https://citizen.some.domain/v1/modules/eric/rds-oracle/aws/0.1.0/download
2021-05-09T19:18:57.144+0200 [TRACE] ModuleInstaller: eric citizen.some.domain/eric/rds-oracle/aws 0.1.0 is available at "https://citizen.some.domain/v1/modules/tarball/eric/rds-oracle/aws/0.1.0/module.tar.gz"
2021-05-09T19:18:57.144+0200 [DEBUG] will download "https://citizen.some.domain/v1/modules/tarball/eric/rds-oracle/aws/0.1.0/module.tar.gz" to .terraform\modules\eric
2021-05-09T19:18:57.144+0200 [TRACE] fetching "https://citizen.some.domain/v1/modules/tarball/eric/rds-oracle/aws/0.1.0/module.tar.gz" to ".terraform\\modules\\eric"
2021-05-09T19:18:57.380+0200 [TRACE] modsdir: writing modules manifest to .terraform\modules\modules.json
Crash Output
N/A
Expected Behavior
Send an Authorization header on every request to the private registry
Actual Behavior
The call to https://citizen.some.domain/v1/modules/tarball/eric/rds-oracle/aws/0.1.0/module.tar.gz misses the Authorization header
Steps to Reproduce
-
terraform init
The developers of gitlab have the same issue, see https://gitlab.com/gitlab-org/gitlab/-/issues/321102#note_525700572
Hi @ericrichtert,
What you've described here is, unfortunately, the intended behavior. The reason is that in Terraform's architecture the idea of a "module registry" is an index separate from the actually-published modules, and so the registry authentication only applies to the registry index lookup. The result of a registry lookup is then a real physical source location, which Terraform then handles the same way as if the user had just written that address directly into source, as documented in Module Sources.
In your case it seems like you've made the registry return an https: URL on its own server, and thus the registry is also a distribution point for packages. But Terraform is handling that as described in Module Sources: HTTP URLs, which is not a part of the registry mechanism and does authentication in a different way, via a .netrc file.
Your use-case of having a private registry host its own packages does seem like a reasonable one that we can consider how best to solve. We likely would not do it by changing the behavior of the existing mechanism but rather by introducing some new opt-in mechanism that allows a registry to be explicit that it's intending to self-host and thus that it's intending to pass on its same credentials to the target.
In the meantime, with Terraform as it exists today, I think you would need to have your "download" endpoint return a URL which itself contains some sort of time-limited bearer token in the path or query string to verify the download. For what it's worth, I believe that's how Terraform Cloud's private module registry implements a similar idea for modules hosted within Terraform Cloud itself: the registry and the distribution service are separate systems but the registry is allowed to sign an approval for the caller to download the package, which the distribution service then checks and honors. (I may have the details not quite right here because I don't work on Terraform Cloud, but this is at least roughly the architecture.)
The mention already created a link above, but just to make it extra obvious to a future reader I want to note that we also have #29349 discussing the corresponding potential new feature for provider registries, and so it'd be nice to see if we can define a similar mechanism for both use-cases.
@apparentlymart are there any updates on this?
My organization would like to set up an http-based private module registry, but the lack of [Bearer token] authentication is causing problems.
Perhaps terraform look for a specific environment variable containing the TOKEN and/or acting as a feature toggle to pass the token in the Bearer header.
Hi @johnknutsonhc,
I longer work on Terraform at HashiCorp, so I'm afraid I don't know what the plans are for this feature request.