Investigate: Avoid Starting Provider Cache Server When it Doesn't Help

[yhakbar]

Investigate the advantage to not starting the provider cache server when it doesn't need to be started.

For certain commands like render-json, it slows things down without providing much value.

Let's look into the commands that require the provider cache server, and selectively not start it when it's not needed.

[yhakbar]

Another thing to evaluate, based on conversations in #3363 is not spinning up the provider cache server, but merely setting the appropriate cache directory when necessary for terragrunt apply but not terragrunt run-all apply, etc.

There are implications for what will happen when the unit running terragrunt apply has a dependency that fetches the same providers, though.

[mwos-sl]

Adding show command to the list of the ones that do not need cache

[mwos-sl]

The apply <path to plan> is interesting too. I don't see any provider download operation when i run this, when the plan is stored already on disk.

[yhakbar]

To be clear on the priority of this issue: We're currently not prioritizing it, as we're working on advancing Terragrunt on the road to 1.0. This is an optimization opportunity for the Provider Cache Server that is worth tackling, but requires a lot of testing to confirm that it's a good change, and documentation for verification that it behaves as expected.

If someone in the community is willing to take on the burden of verifying (via Golang tests) that there are certain scenarios where Terragrunt can safely avoid starting up the provider cache server and introduce the logic to prevent it from starting up in those scenarios, we would be happy to take a look at those PRs.

If not, we will have to revisit this when we have down time.

[wosiu]

My 2c: ideally this should be solved on terraform and opentofu end, so that we don't need this extra caching proxy on a terragrunt layer. Starting a service will be always an extra overhead. Related context:

1. opentofu:

• ticket: https://github.com/opentofu/opentofu/issues/1483,
• rfc: https://github.com/opentofu/opentofu/pull/1939

2. terraform:

• issue: issue: https://github.com/hashicorp/terraform/issues/31964,
• PR1: https://github.com/hashicorp/terraform/pull/33479,
• PR2 closer to be merged: https://github.com/hashicorp/terraform/pull/36085