hashicorp
terraform `providers lock` should support plugin_cache_dir
Terraform Version
Terraform v1.3.9
on linux_arm64
Use Cases
We currently manage a "single-tenant" cloud application with a separate tenant for each of our customers. Each tenant is it's own AWS account and terraform state, and each tenant pulls in a handful of modules we've written to configure the infrastructure for our application. When we make a change to our application, we run terragrunt run-all apply to make this change to all of the tenants at the same time.
We'd like to keep our terraform providers up to date, utilizing caches to prevent doing the same thing over and over.
Attempted Solutions
To reduce traffic and speed up run-times, we take advantage of plugin_cache_dir so that each tenant pulls files from the local cache instead of retrieving it from the network.
When new versions of providers come out, we upgrade. e.g. when the aws provider 4.56.0 gets bumped to 4.57.0 we upgrade versions. To do this, we upgrade a single tenant with all of the terraform providers (because the cache dir isn't multi-terraform-instance safe for writing!), then run terragrunt run-all init -upgrade. This uses the cache and works great.
Because of the mix of x86 and arm64, we also need to run terragrunt run-all providers lock -platform=linux_arm64 -platform=linux_amd64. (and we are waiting on Terraform v1.4 which should prevent the need for running this). This appears to ignore plugin_cache_dir and downloads everything, again, for every tenant:
[...]
Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.
- Fetching hashicorp/aws 4.57.0 for linux_arm64...
- Retrieved hashicorp/aws 4.57.0 for linux_arm64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.3.0 for linux_arm64...
- Retrieved hashicorp/archive 2.3.0 for linux_arm64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.3.0 for linux_amd64...
- Retrieved hashicorp/archive 2.3.0 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/aws 4.57.0 for linux_amd64...
- Retrieved hashicorp/aws 4.57.0 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/aws checksums for linux_arm64; Additional checksums for this platform are now tracked in the lock file
- Obtained hashicorp/aws checksums for linux_amd64; Additional checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for linux_arm64; Additional checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for linux_amd64; Additional checksums for this platform are now tracked in the lock file
Success! Terraform has updated the lock file.
Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.
- Fetching hashicorp/aws 4.57.0 for linux_arm64...
- Retrieved hashicorp/aws 4.57.0 for linux_arm64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.3.0 for linux_arm64...
- Retrieved hashicorp/archive 2.3.0 for linux_arm64 (signed by HashiCorp)
- Fetching hashicorp/aws 4.57.0 for linux_amd64...
- Retrieved hashicorp/aws 4.57.0 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.3.0 for linux_amd64...
- Retrieved hashicorp/archive 2.3.0 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/aws checksums for linux_arm64; All checksums for this platform were already tracked in the lock file
- Obtained hashicorp/aws checksums for linux_amd64; Additional checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for linux_arm64; All checksums for this platform were already tracked in the lock file
- Obtained hashicorp/archive checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
Success! Terraform has updated the lock file.
[...]
Proposal
It would be great for providers lock to use plugin_cache_dir. This would make keeping providers up to date in a multi-platform environment much quicker and easier, and more pleasant. Thanks!
References
No response
Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!
Terraform 1.4 is here!
Terraform v1.4.0
on linux_arm64
I don't think this has resolved the providers lock being required for multi-architecture environments yet as running providers lock with clean lock files here still adds new entries for the arch different than my native arch.
And it's still slow because it's fetching everything again/still.
As I understand it, per https://github.com/hashicorp/terraform/pull/32129 providers lock will always ignore the cache the first time it is run - is this request to change that behavior?
That does appear to be the case for a first-run of providers lock for a single terraform context, and it seems completely right that the first run of providers lock using a cache that doesn't have what it needs would have to query the origin.
The thing that doesn't seem right here is that when a terraform context (lets call it 'A') has run providers lock and snagged the hash/providers for aws and archive, when a different terraform context (lets call it 'B') runs providers lock and is looking for aws and archive, B should be able to utilize what is in the cache without it having to fetch everything from the origin.
+100 Nothing more infuriating than having to rebuild lock files in a thousand stacks
