Feature: Tofu providers cache

[Despire]

Motivation

To speed up the building of the cluster, we can shave off a few seconds, or even up-to some minutes in maintaining a tofu providers plugin cache.

Description

Explore if the env TF_PLUGIN_CACHE_DIR= is suitable to implement a provider plugin cache so that they do not need to be re-downloaded every-time a change happens inside the cluster.

Exit criteria

• [ ] explore if TF_PLUGIN_CACHE_DIR= can be used for implementing a provider plugin cache

[Despire]

TF_PLUGIN_CACHE_DIR= is not suitable due to

Note: The plugin cache directory is not guaranteed to be concurrency safe. The provider installer's behavior in environments with multiple terraform init calls is undefined.

https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache

Also we need to persists the .terraform.lock.hcl file for the cache to work which we currently do not do in claudie. On each workflow run it gets regenerated and then deleted.

[Despire]

we could work around the concurrency limit of TF_PLUGIN_CACHE_DIR= by having each cluster have its own plugin cache

[bernardhalas]

Could you write a design doc for this feature request pls?

[m-brando]

Design doc for the issue : https://docs.google.com/document/d/1qv0YXomaHSEMfOZKGEv_UqC69Of0LREM8S5RI9JcUyQ/edit?tab=t.0#heading=h.ol1douhhte9n