terraform
terraform copied to clipboard
hashicorp
Not clear how to configure JFrog Artifactory as a provider network mirror (and whether Artifactory supports that protocol)
Terraform Version
Terraform v1.2.9
on windows_amd64
Terraform Configuration Files
terraform.rc
provider_installation {
network_mirror {
url = "https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/"
}
}
providers.tf
terraform {
required_providers {
azurerm = {
source = "providers/hashicorp/azurerm"
version = "~> 3.14.0"
}
random = {
source = "providers/hashicorp/random"
version = "~> 3.3.0"
}
}
}
provider "azurerm" {
features {
resource_group {
prevent_deletion_if_contains_resources = false
}
}
}
provider "random" {}
terraform {
required_version = "~> 1.2.6"
}
Debug Output
terraform init
Initializing provider plugins...
- Finding providers/hashicorp/azurerm versions matching "~> 3.14.0"...
2022-09-16T13:45:02.754+0100 [DEBUG] Querying available versions of provider providers/hashicorp/azurerm at network mirror https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
2022-09-16T13:45:02.755+0100 [DEBUG] GET https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/azurerm/index.json
2022-09-16T13:45:02.755+0100 [TRACE] HTTP client GET request to https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/azurerm/index.json
- Finding providers/hashicorp/random versions matching "~> 3.3.0"...
2022-09-16T13:45:02.956+0100 [DEBUG] Querying available versions of provider providers/hashicorp/random at network mirror https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
2022-09-16T13:45:02.957+0100 [DEBUG] GET https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/random/index.json
2022-09-16T13:45:02.961+0100 [TRACE] HTTP client GET request to https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/random/index.json
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider providers/hashicorp/azurerm: provider providers/hashicorp/azurerm was not found in any of the search
│ locations
│
│ - provider mirror at https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
╵
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider providers/hashicorp/random: provider providers/hashicorp/random was not found in any of the search
│ locations
│
│ - provider mirror at https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
Expected Behavior
I would of expected /versions to be queried (not /index.json).
expected version checking urls:
- https://packages.redacteddomainname..com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/azurerm/versions
- https://packages.redacteddomainname..com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/random/versions
the following query works
curl 'https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/random/versions'
the example below is without the 'network_mirror' configured. as you can see /versions is queried. which works with the mirror e.g.
Initializing provider plugins...
- Reusing previous version of hashicorp/azurerm from the dependency lock file
2022-09-16T14:24:41.769+0100 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2022-09-16T14:24:41.787+0100 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2022-09-16T14:24:42.313+0100 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
2022-09-16T14:24:42.315+0100 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
- Reusing previous version of hashicorp/random from the dependency lock file
2022-09-16T14:24:42.455+0100 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/random/versions
2022-09-16T14:24:42.455+0100 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/random/versions
2022-09-16T14:24:42.551+0100 [TRACE] providercache.fillMetaCache: scanning directory .terraform\providers
2022-09-16T14:24:42.560+0100 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform\providers: CreateFile .terraform: The system cannot find the file specified.
2022-09-16T14:24:42.569+0100 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform\providers: cannot search .terraform\providers: CreateFile .terraform\providers: The system cannot find the path specified.
2022-09-16T14:24:42.573+0100 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/azurerm/3.14.0/download/windows/amd64
2022-09-16T14:24:42.573+0100 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/azurerm/3.14.0/download/windows/amd64
2022-09-16T14:24:42.652+0100 [DEBUG] GET https://releases.hashicorp.com/terraform-provider-azurerm/3.14.0/terraform-provider-azurerm_3.14.0_SHA256SUMS
2022-09-16T14:24:42.653+0100 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-azurerm/3.14.0/terraform-provider-azurerm_3.14.0_SHA256SUMS
2022-09-16T14:24:42.705+0100 [DEBUG] GET https://releases.hashicorp.com/terraform-provider-azurerm/3.14.0/terraform-provider-azurerm_3.14.0_SHA256SUMS.72D7468F.sig
2022-09-16T14:24:42.707+0100 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-azurerm/3.14.0/terraform-provider-azurerm_3.14.0_SHA256SUMS.72D7468F.sig
- Installing hashicorp/azurerm v3.14.0...
2022-09-16T14:24:42.798+0100 [TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/hashicorp/azurerm v3.14.0 from https://releases.hashicorp.com/terraform-provider-azurerm/3.14.0/terraform-provider-azurerm_3.14.0_windows_amd64.zip
Actual Behavior
terraform init
2022-09-16T14:12:04.069+0100 [INFO] Terraform version: 1.2.9
2022-09-16T14:12:04.070+0100 [DEBUG] using github.com/hashicorp/go-tfe v1.0.0
2022-09-16T14:12:04.071+0100 [DEBUG] using github.com/hashicorp/hcl/v2 v2.12.0
2022-09-16T14:12:04.071+0100 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2022-09-16T14:12:04.071+0100 [DEBUG] using github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
2022-09-16T14:12:04.071+0100 [DEBUG] using github.com/zclconf/go-cty v1.11.0
2022-09-16T14:12:04.071+0100 [INFO] Go runtime version: go1.18.1
2022-09-16T14:12:04.071+0100 [INFO] CLI args: []string{"C:\\util\\terraform.exe", "init"}
2022-09-16T14:12:04.072+0100 [TRACE] Stdout is a terminal of width 167
2022-09-16T14:12:04.072+0100 [TRACE] Stderr is a terminal of width 167
2022-09-16T14:12:04.073+0100 [TRACE] Stdin is a terminal
2022-09-16T14:12:04.087+0100 [DEBUG] Attempting to open CLI config file: C:\Users\redactedusername\AppData\Roaming\terraform.rc
2022-09-16T14:12:04.147+0100 [INFO] Loading CLI configuration from C:\Users\redactedusername\AppData\Roaming\terraform.rc
2022-09-16T14:12:04.161+0100 [DEBUG] Explicit provider installation configuration is set
2022-09-16T14:12:04.166+0100 [TRACE] Selected provider installation method cliconfig.ProviderInstallationNetworkMirror("https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/") with includes [] and excludes []
2022-09-16T14:12:04.167+0100 [INFO] CLI command args: []string{"init"}
Initializing the backend...
2022-09-16T14:12:04.218+0100 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2022-09-16T14:12:04.223+0100 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2022-09-16T14:12:04.223+0100 [DEBUG] New state was assigned lineage "7799abfe-048b-eb05-f727-9c2dc0fbe538"
2022-09-16T14:12:04.223+0100 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2022-09-16T14:12:04.224+0100 [TRACE] Meta.Backend: instantiated backend of type <nil>
2022-09-16T14:12:04.231+0100 [TRACE] providercache.fillMetaCache: scanning directory .terraform\providers
2022-09-16T14:12:04.233+0100 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform\providers: CreateFile .terraform: The system cannot find the file specified.
2022-09-16T14:12:04.234+0100 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform\providers: cannot search .terraform\providers: CreateFile .terraform\providers: The system cannot find the path specified.
2022-09-16T14:12:04.234+0100 [TRACE] providercache.fillMetaCache: scanning directory .terraform\providers
2022-09-16T14:12:04.235+0100 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform\providers: CreateFile .terraform: The system cannot find the file specified.
2022-09-16T14:12:04.272+0100 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform\providers: cannot search .terraform\providers: CreateFile .terraform\providers: The system cannot find the path specified.
2022-09-16T14:12:04.296+0100 [DEBUG] checking for provisioner in "."
2022-09-16T14:12:04.307+0100 [DEBUG] checking for provisioner in "C:\\util"
2022-09-16T14:12:04.310+0100 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2022-09-16T14:12:04.316+0100 [TRACE] backend/local: state manager for workspace "default" will:
- read initial snapshot from terraform.tfstate
- write new snapshots to terraform.tfstate
- create any backup at terraform.tfstate.backup
2022-09-16T14:12:04.319+0100 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2022-09-16T14:12:04.324+0100 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2022-09-16T14:12:04.451+0100 [TRACE] statemgr.Filesystem: read nil snapshot
Initializing provider plugins...
- Finding providers/hashicorp/azurerm versions matching "~> 3.14.0"...
2022-09-16T14:12:04.563+0100 [DEBUG] Querying available versions of provider providers/hashicorp/azurerm at network mirror https://packages.redacteddomainnamecom/artifactory/hashicorp-registry-remote/v1/
2022-09-16T14:12:04.582+0100 [DEBUG] GET https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/azurerm/index.json
2022-09-16T14:12:04.601+0100 [TRACE] HTTP client GET request to https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/azurerm/index.json
- Finding providers/hashicorp/random versions matching "~> 3.3.0"...
2022-09-16T14:12:05.471+0100 [DEBUG] Querying available versions of provider providers/hashicorp/random at network mirror https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
2022-09-16T14:12:05.473+0100 [DEBUG] GET https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/random/index.json
2022-09-16T14:12:05.473+0100 [TRACE] HTTP client GET request to https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/providers/hashicorp/random/index.json
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider providers/hashicorp/azurerm: provider providers/hashicorp/azurerm was not found in any of the search
│ locations
│
│ - provider mirror at https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
╵
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider providers/hashicorp/random: provider providers/hashicorp/random was not found in any of the search
│ locations
│
│ - provider mirror at https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/
Steps to Reproduce
terraform init
Additional Context
i'm attempting to configure a network_mirror.
for some reason, the version check is failing due to /index.json being queried, not /versions (which is what is called without the mirror configured).
References
No response
Hi @Fodsuk,
From what you are describing, it seems like you have created a provider registry but you are attempting to use it in Terraform as if it were a provider mirror.
These two concepts have different meanings in Terraform:
- A provider registry is the authoritative source for some providers. Its hostname becomes part of their unique addresses. A registry only serves providers whose addresses include its hostname.
- A provider mirror is so optional alternative source for providers that could potentially have originated in many different registries. Because a network mirror can serve providers that belong to many different registry hostnames, its wire protocol is different to specify which hostname the mirror is being asked to act on behalf of.
You mention the hashicorp/azurerm provider, whose canonical name is registry.terraform.io/hashicorp/terraform and so its origin registry is registry.terraform.io. If you want to serve that provider from a server under your control then a network mirror is the correct way to do it but the mirror will need to implement the network mirror protocol instead of the registry protocol in order to do so.
A server which implements the registry protocol can only serve providers under its own hostname, like packages.redacteddomainname.com/foo/bar. It cannot serve packages for providers that belong to the registry.terraform.io namespaces.
Hi @apparentlymart
you're absolutely right. Due to company policy, we can't access registry.terraform.io directly.
In the meantime an Artifactory mirror (https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/) has been created for https://registry.terraform.io/. The intention is Terraform would be configured to use the mirror to download common providers (azurerm, random etc...)..
Looking at the Network Mirror Protocol documentation
terraform.rc configured:
provider_installation {
network_mirror {
url = "https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/"
}
}
provider.tf configured:
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 3.14.0"
}
random = {
source = "hashicorp/random"
version = "~> 3.3.0"
}
}
}
upon executing terraform init.
the provider's hostname, namespace and type is appended to the download URL. Obviously this is an incorrect URL. Am i missing something?
Hi @Fodsuk!
Those URLs in the log you shared seem correct to me. For the benefit of anyone who can't see the image, the two URLs shown in the log are (with the hostname redacted):
GET https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/registry.terraform.io/hashicorp/random/index.json
GET https://packages.redacteddomainname.com/artifactory/hashicorp-registry-remote/v1/registry.terraform.io/hashicorp/azurerm/index.json
This matches the structure of the List Available Versions operation. Terraform CLI is asking your mirror to enumerate which versions it has available for the registry.terraform.io/hashicorp/random and registry.terraform.io/hashicorp/azurerm providers.
I'm afraid I'm not familiar with Artifactory's implementation of the provider mirror protocol (I don't have an Artifactory server to test with) so I'm not sure what to suggest to get it working. If you are able, it may help to ask JFrog support whether support for the provider mirror protocol is available and, if so, how to set it up. I think at the moment any of the following could be true from what I can see:
- The registry configured in Artifactory is a provider origin registry rather than a provider mirror. Though we already discussed that and it sounds like you've replaced the origin registry with a mirror now.
- Artifactory does implement the mirror protocol but it requires you to set a different base URL in the configuration to make it work. I'm not familiar with Artifactory's URL scheme so I'm not meaning to suggest the URL you showed is definitely wrong, just that I'm unable to confirm whether it's correct.
- Artifactory doesn't implement the mirror protocol, and so there isn't any automatic way to set this up. If this is true then you may need to "implement the protocol" using a generic registry containing manually-generated JSON index files and packages, similar to what
terraform providers mirrorwould generate on your local disk but hosted in Artifactory instead.
If you do ask JFrog support about this and learn more then I'd love to hear what you learn. We did recently talk about this over in #31624 and someone else said they'd found Artifactory to support the mirror protocol through local testing, but if we can then I'd love to hear what JFrog themselves have to say about which of Terraform's protocols Artifactory is currently intended to support.
I've been trying to configure terraform to use artifactory terrafrom provider registry with no luck. Seems terraform only supports hostname/namespace/providername and the path to the provider registry in our artifactory instance is artifactory.acme.com/artifactory/terraform-provider-local/eng/bitbucketserver. Terraform complains that the path is not valid.
The "source" attribute must be in the format "[hostname/][namespace/]name"
terraform {
required_providers {
bitbucketserver = {
source = "artifactory.acme.com/artifactory/terraform-provider-local/eng/bitbucketserver"
version = "1.6.3"
}
}
}
provider "bitbucketserver" {
# Configuration options
}
Hi @ryancurrah,
Since you are using your Artifactory hostname as part of the address of your provider that means you are using Artifactory's implementation of the provider registry protocol, and not the provider network mirror protocol. The provider mirror protocol is what allows you to serve a provider from "the wrong hostname" (that is: from somewhere other than the hostname embedded in its source address).
If Artifactory has a correct implementation of the provider registry protocol then it should work with a three-part provider source address whose first part is your Artifactory hostname. I don't know enough about Artifactory's implementation of this protocol to know what belongs in the namespace and name components; the public documentation doesn't show any examples of how to write a required_providers entry for a provider hosted in an Artifactory provider registry. Since you are an Artifactory customer you can hopefully get some guidance from JFrog technical support on that.
This issue is talking about the provider network mirror protocol rather than the provider registry protocol, so what you tried there is not relevant to this particular issue. However, if you do get more information on how to use Artifactory's implementation of the registry protocol then I'd like to learn more about it so I can more confidently help others in the future; if you learn something that you'd be willing to share I'd love if you could start a topic in the community forum about it.
Hey @apparentlymart thanks for the reply. I have opened a support ticket with jFrog to figure it out. I see they do implement the Service Discovery protocol but do not advertise a providers.v1.
Service Discovery response from running a trial of artifactory pro locally, http://artifactory.crwd.local:8082/.well-known/terraform.json.
{
"modules.v1": "http://artifactory.crwd.local:8082/artifactory/api/terraform/v1/modules/",
"state.v2": "http://artifactory.crwd.local:8082/artifactory/api/terraform/remote/v2",
"tfe.v2": "http://artifactory.crwd.local:8082/artifactory/api/terraform/remote/v2",
"tfe.v2.1": "http://artifactory.crwd.local:8082/artifactory/api/terraform/remote/v2",
"tfe.v2.2": "http://artifactory.crwd.local:8082/artifactory/api/terraform/remote/v2",
"login.v1": {
"client": "terraform-cli",
"authz": "http://artifactory.crwd.local/ui/terraform/oauth2/authorize",
"token": "http://artifactory.crwd.local:8082/artifactory/api/oauth2/token",
"grant_types": [
"authz_code"
]
}
}
I will work with them to figure it out add a topic on the community forum.
We eventually got this working. Turns out, it was an issue with the version of jfrog, we upgraded to 7.46.11 (7.33 was an issue).
Upon following this guide we were able to set up our terraform.rc
provider_installation {
direct {
exclude = ["registry.terraform.io/*/*"]
}
network_mirror {
url = "https://packages.redacted.com/artifactory/api/terraform/terraform/providers/"
}
}
I got the same error, I achieve to "terraform init" with this message at the end :
Installed <NAMESPACE>/<PROVIDER> v0.0.1 (unauthenticated)
My provider is well downloaded but whenever I add a single line into my main.tf, I got this error :
On TF Init
Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/<NAMESPACE>: provider registry.terraform.io/hashicorp/<NAMESPACE> was not found in any of the search locations
│
│ - provider mirror at https://<MY_ARTIFACTORY_URL>/artifactory/api/terraform/<MY_REPOSITORY>/providers/
On TF Plan
╷
│ Error: Inconsistent dependency lock file
│
│ The following dependency selections recorded in the lock file are inconsistent with the current configuration:
│ - provider registry.terraform.io/hashicorp/<MY_NAMESPACE>: required by this configuration but no version is selected
│
│ To update the locked dependency selections to match a changed configuration, run:
│ terraform init -upgrade
╵
I don't know why "hashicorp" is there on my provider as I never mentioned it before.
Hi all,
I think there are a few different remaining confusions here which hopefully we're getting close to clearing up.
The first is that the example above of using a network_mirror block does suggest that Artifactory is able to support the provider mirror protocol, as opposed to the provider registry protocol. That means it's feasible to use Artifactory to host a local mirror of providers that would normally come from some other registry such as registry.terraform.io.
The documentation page seems a little unclear about what exactly it is describing, though. I see it talk about the path layout for a provider registry rather than a provider mirror and so I'm not sure how those instructions exactly can lead to having a network mirror. @Fodsuk did you need to populate the Artifactory repository in a different way than shown on that page, such as including a directory called registry.terraform.io so that Artifactory can see which of the packages are mirrors of packages from the official public registry?
The other separate confusion seems to be about using Artifactory to host in-house providers that are not normally published on registry.terraform.io. In that case you will need to select a domain under your control to be your private namespace of providers, completely separate from the ones in the public registry. When you use these in a Terraform module, each module that uses them must declare the full source address of the module in the required_providers block. By default Terraform assumes that undeclared providers are intended to be official providers for backward compatibility with modules written before Terraform supported third-party providers. If you see Terraform trying to install something from the hashicorp namespace that doesn't belong to that namespace then that means at least one of your modules is missing its declaration that it requires your custom provider.
jFrog says when you deploy an artifact to the tf provider registry in Artifactory you MUST include a namespace.
So it looks like this in Artifactory...
Then your provider configuration must include the namespace (Don't forget to create the terraform.rc file).
terraform {
required_providers {
azurerm = {
source = "testnamespace/bitbucketserver"
version = "1.6.3"
}
}
}
One caveat is that Artifactory does not provide the shasums, in a way it doesn't fully support the provider mirror protocol. They have an open internal ticket tracking work to support it.
We have an open internal Jira regarding this issue - RTDEV-27570.
Artifactory Terraform provider registry respects the Provider Network Mirror Protocol: https://www.terraform.io/internals/provider-network-mirror-protocol
Currently, it is missing the hashes field. The missing hash, cause an "(unauthenticated)" message when installing providers through Artifactory:
- Installed hashicorp/null v3.1.1 (unauthenticated)
While the expected installation flow with the hash should print:
- Installed hashicorp/null v3.1.1 (verified checksum)
Artifactory should be able to calculate the required hash and return it in its response.
My last response is pretty much a copy paste of @apparentlymart last comment. In an attempt to get more information about the inner-workings of Artifactories implementation.
If I understand well @ryancurrah , we have to wait this issue to be fix by Artifactory ? Because what you've done is exactly what I have done
No you do not have to wait for that feature, I got it working today. We needed to setup a terraform local, remote and virtual repository in Artifactory. The local repository is where we publish our providers, the remote repository proxies registry.terraform.io/*/* and finally the virtual merges the local and remote repositories so they look like one repository.
So my terraformrc file config looks like.
╰─ cat ~/.terraformrc
provider_installation {
direct {
exclude = ["registry.terraform.io/*/*"]
}
network_mirror {
url = "https://artifactory.acme.com/artifactory/api/terraform/terraform-prod-virtual/providers/"
}
}
I got another response from jFrog yesterday.
Hi Ryan,
I’d like to clarify that Artifactory currently only supports the provider network mirror protocol and is not intended to serve > as an origin registry which would enable browsing providers and documentation for the same, etc in addition to hosting the actual providers.
The local providers registry is meant to resolve a customer’s in-house providers from Artifactory every time the customer performs terraform init.
You can also take a look at our YouTube video here to learn more about how Terraform works with Artifactory (from around minute 8).
I hope the above information helps clarify.
So it looks like Artifactory only supports the provider network mirror protocol (not fully compliant because it does not send the hashes).
Hey @ryancurrah I got exactly the same configuration as you and it does not work. I can init but when I add one resource in my main.tf I got error with my provider :/
Ok, it seems that un update on my Artifactory solved the issue. But now I'm facing another issue. Is it possible to use my provider hosted on artifactory AND to use another provider from another registry (like hashicorp/aws from the terraform registry) ? With the terraformrc that excludes the terraform registry from hashicorp, i can't get the AWS provider. Any idea ?
Terraform has two different ways to install providers over the network, which are called direct and network_mirror in the CLI configuration's provider_installation block:
-
directmeans that the provider will be installed directly from its origin registry. The "origin registry" is the hostname that is part of the source address. For example, intf.example.com/foo/barthe origin registry istf.example.com. A provider address with only two parts, likehashicorp/aws, is really just a shorthand forregistry.terraform.io/hashicorp/awsand so its origin registry isregistry.terraform.io. -
network_mirrorallows establishing an alternative location for any number of providers that can each belong to any origin registry. This allows severing the connection between the namespaceregistry.terraform.io/and the physical service that hostname provides; the provider mirror protocol includes the origin registry hostname as part of all requests so that the mirror can distinguish between potentially many different origin registries that it is mirroring for.
From what I can tell from the discussion so far, it seems that Artifactory can be a server for either or both of these modes. I expect the simplest path for most people would be to set up Artifactory as a network mirror and copy all of the providers you intend to use in the mirror, and then use a relatively simple CLI configuration which specifies only the network mirror and disables (by omission) the "direct" installation method:
provider_installation {
network_mirror {
url = "https://artifactory.example.com/artifactory/api/terraform/terraform-prod-virtual/providers/"
}
}
However, Terraform does allow arbitrary combinations of the installation methods if you'd like to do something more elaborate.
For example, if you intend to use Artifactory only to host providers you've developed in-house and wish to continue to use the public Terraform Registry for the official and community-maintained providers then I'd typically recommend to set Artifactory up as a provider registry rather than a provider mirror and then leave the CLI configuration selecting only the direct installation method for all providers:
# (this is essentially the default, as long as you don't have any
# providers on your local system in the implied mirror directories)
provider_installation {
direct {}
}
In this direct-only approach the idea would be to make your in-house providers have an "origin registry" hostname that is in a domain that belongs to your organization. I've been using example.com as a placeholder in the discussion so far, but of course you should use a hostname you actually control rather than that placeholder. You can make any arbitrary hostname you control behave as an origin registry by adding a remote service discovery document to a HTTPS server running at that hostname, which can then use an endpoint in your Artifactory as the actual provider registry API -- the hostname with the discovery document does not necessarily need to match the host where Artifactory is running:
{
"providers.v1": "https://artifactory.example.com/artifactory/something/something/"
}
You'd then select providers from your Artifactory-based registry by including your origin registry hostname in the provider source addresses: source = "terraform.example.com/foo/bar" if you published the discovery document on the host terraform.example.com.
I believe from earlier discussion that Artifactory does include a Terraform service discovery document on its own server. If that's true and if it includes a "providers.v1" entry like what I showed above then you would be able to use your Artifactory hostname as the origin registry hostname and not need to set up another hostname, although this does mean that your Artifactory hostname will be hard-coded into each module that uses one of your in-house providers. Setting up a separate host allows an extra level of indirection in case you e.g. need to move Artifactory to a different hostname later, or if you choose to stop using Artifactory and use a different provider registry implementation instead.
Finally, you can mix these models together by writing a more complicated CLI configuration. I think this particular example is the one that matches the most recent question above about mixing some providers mirrored in Artifactory while some other providers are coming from their origin registries.
You can achieve that by including multiple installation method blocks in the provider_installation container, and using include and exclude arguments on each one to explain to Terraform which installation methods to use for which providers:
provider_installation {
network_mirror {
url = "https://artifactory.example.com/artifactory/api/terraform/terraform-prod-virtual/providers/"
include = ["registry.terraform.io/hashicorp/null"]
}
direct {
exclude = ["registry.terraform.io/hashicorp/null"]
}
}
This more complicated example tells Terraform that it should install hashicorp/null (for example) from a mirror in Artifactory, but it should install all other providers from their origin registries. This would be a reasonable configuration if, for example, you've temporarily forked an official provider to incorporate a change that hasn't been released officially yet, but you still want your modules to think of your fork as being registry.terraform.io/hashicorp/null, rather than some other address in your own namespace.
The include and exclude arguments support wildcards in the rightmost positions, so you can also include and exclude entire namespaces within a particular registry host or entire registry hosts:
include = ["registry.terraform.io/hashicorp/*"]
include = ["registry.terraform.io/*/*"]
I would typically recommend keeping things simple by using only a single installation source and then using server-side configuration (service discovery documents and provider mirrors) to encapsulate the complexity of where exactly the providers are coming from, so that you can change things centrally if you need to.
Terraform does support more complex client-side configurations for situations where it's unavoidable, though.
Not sure if this belongs in this thread as there were many references to hosting the files in the mirror. In my case, I am just interested in getting a remote repository working, where the artifactory url directs to https://registry.terraform.io. Testing different variations of this config didnt seem to work
provider_installation {
direct {
exclude = ["registry.terraform.io/*/*"]
}
network_mirror {
url = "https://artifactoryBaseUrl/artifactory/remote-repo-key/"
}
And would result in this error
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/random: failed to query provider mirror
│ https://artifactoryBaseUrl/artifactory/remote-repo-key/ for
│ registry.terraform.io/hashicorp/random: invalid response content from
│ mirror server: invalid character '<' looking for beginning of value
