Feature: Move provider versions to external template repository

[Despire]

Motivation

Currently the provider versions are part of claudie itself, resulting in the claudie version dictating what provider version is to be used for the external templates. Ideally this would be part of the external templates itself so that they also control the version of the supported providers.

Description

Move https://github.com/berops/claudie/blob/master/services/terraformer/server/domain/utils/templates/providers.tpl into https://github.com/berops/claudie-config to have the templates also specify the versions of the provider for terraform, while also keeping it inside claudie for backwards compatibility

Exit criteria

• [ ] Add https://github.com/berops/claudie/blob/master/services/terraformer/server/domain/utils/templates/providers.tpl into https://github.com/berops/claudie-config

• [ ] https://github.com/berops/claudie/blob/82389fb6faac3e052d2b38f76da3277125b40ed8/services/terraformer/server/domain/utils/templates/provider.go#L21 move into and properly document it https://github.com/berops/claudie/blob/master/services/terraformer/server/domain/utils/templates/structures.go

• [ ] Adjust the generation of the templates inside terraformer, to prefer using the versions from the external template repository, if present, if not default to using the one inside claudie.

NOTE: for backwards compatibility, the providers.tpl should stay inside claudie but it should also look at the external templates and if its present there it should prefer using that one, For now we need to make it part of claudie for backwards compatibility, but it will be removed in future versions

[m-brando]

Some progress has been made in the following branch: https://github.com/m-brando/claudie/tree/feature/external-providers-template

• Users can now define a required_providers block in a custom file called required_providers (NOTE file without the .tpl suffix).
• The .tpl suffix is intentionally avoided to prevent the file from being automatically merged into terraformer/server/cluster, which would cause conflicts with terraformer/server/domain/utils/templates/providers.tpl. Instead, the file is parsed manually, and its contents are selectively merged into providers.tpl.

Example content of the custom required_providers file:

google = {
  source  = "hashicorp/google"
  version = "6.45.0"
}

Current Issue When a user updates the provider version in the required_providers file, a version conflict occur:

brando-test-u2blggy	 │ Error: Failed to resolve provider packages
brando-test-u2blggy	 │ 
brando-test-u2blggy	 │ Could not resolve provider hashicorp/google: locked provider
brando-test-u2blggy	 │ registry.opentofu.org/hashicorp/google 6.43.0 does not match configured
brando-test-u2blggy	 │ version constraint 6.45.0; must use tofu init -upgrade to allow selection
brando-test-u2blggy	 │ of new versions
brando-test-u2blggy	 ╵

This happens because the lock file (from .terraform.lock.hcl) expects version 6.43.0, but the user-defined config specifies 6.45.0.

Things to discuss with @Despire

• How to decide which provider to use for current vs desired cluster
• Automation around running tofu init -upgrade when needed.

[Despire]

The main issue blocking this is as follows.

Say you have a template version of v0.9.8 which has hetzner v6.0.0. Next you want to upgrade to the template version v0.10.0 which has hetzner v7.0.0..

This will end up with collisions because of the way claudie works.

• We have a single state file for the whole cluster. If you mix-match or replace a nodepool with the old template version with the new, You will run into collision into which version to use. Which maybe you want the new version, but then since we have a single state file for the whole cluster, all of the other resources that also use the old version and for which the template update was not requested will also end up being updated.

To implement this, we would need to split everything up in claudie into the smallest possible unit that can work idependently, which will be backwards incompatible.

Instead of a single state file, we could have a single state file for each such unit, this way we would implement this versioning within the external templates.