terraform icon indicating copy to clipboard operation
terraform copied to clipboard

Concurrency-safe unpacking of TF providers.

Open pietrodn opened this issue 3 months ago • 5 comments

The original implementation unpacked the downloaded provider .zip files directly in the Terraform plugin cache directory. This made the terraform init command prone to race conditions when multiple Terraform modules using the same cache were downloading, checksumming, and validating the provider files at the same time. This is a serious problem in CI workflows, as it forces developers to choose between two unattractive options:

  1. initializing modules serially and using the cache, or
  2. initializing modules in parallel, but download the same modules every time, increasing the consumed bandwidth.

This change unpacks the .zip files in a temporary directory with a unique name inside the plugin cache directory, and only then moves the files to the expected location.

It also takes a file lock on the target cache directory to prevent two Terraform processes from installing the same provider and conflicting with each other.

This change is a simpler version of this change by @mplzik, rebased on the latest main branch: https://github.com/hashicorp/terraform/pull/33479

Fixes https://github.com/hashicorp/terraform/issues/31964

Target Release

1.11.x

Draft CHANGELOG entry

BUG FIXES

  • Concurrent write access to plugin cache directory has been fixed.

pietrodn avatar Nov 23 '24 12:11 pietrodn