feat: multi module components

[vincenthsh]

Multi Module components

I'm adopting existing Terraform IaC for which Fogg will take over all boilerplate, but in my use-case there are multiple modules per component.

This PR includes all the features I'm adding to support existing code from my fogg fork. It's probably too much to add in a single PR, in which case I'm open to break it down if desired.

Summary

• Add modules[] field to env.component spec to allow multiple modules per component
• Add support for sensitive outputs (copy module output.sensitive value from module to generated tf)
• Add ability to disable global at env config level
• Add support for terraform-provider-sops

Multiple modules per component

To support multiple modules per component, a new field is added to the env.component spec. An array of type componentModule{Source, Name, Variables, Prefix} is added.

• Source, Name and Variables work identical to module_source, module_name and variables from componentSpec
• Prefix is used to separate multiple invocations of the same module, it's added to locals and outputs

Support for sensitive module outputs

Newer versions of Terraform error out if parent passes through module outputs which are sensitive, without flagging them as senstive. Simple fix just copies over the sensitive field from the module invocation

Disable Global at env spec

By default, Fogg adds a remote_state data source for global to each component / environment. For an edge case we isolate sandbox envs so they can't access the global state by design and it was necessary to exclude global state

Add support for terraform-provider-sops

mozilla/sops provides a way to commit secrets together with the terraform code. Ideally secrets are managed directly in the Secrets Management tool (Vault / AWS SecretsManager) and workload are explicitly granted access to their identity to fetch these secrets.

Secrets passed into Terraform configuration through this provider are also persisted in Terraform state, which is not desired at all.

However, the existing code base I'm adopting has password in state file and this provider improves the manual workflow of sharing secrets files.

Test Plan

All existing apply_test.go cases are updated to call the underlying moduleInvocation with multiple modules. One new TestApplyModuleInvocationWithModulePrefix is added to test module generation with Prefix.

Currently the golden test integration was disabled (I did not regenerate testdata), so I would need to re-enable this if we want to merge this.

[jakeyheath]

@vincenthsh is this ready for review?

[vincenthsh]

held up for over a week without any additional changes needed so far, all legacy TF has been adopted by fogg.

[vincenthsh]

Oops, I forgot I have to update testdata and re-enable the ci

[jakeyheath]

Sorry I took so long with this PR. I have a few thoughts that might require some refactoring. Let's pull out each of these features into the following PRs, I think it will make it easier to review and test:

• Support for sensitive module outputs
• Disable Global at env spec
• Provider updates
• Multi-module support (***)

Provider updates

With regards to the sops provider, we have been wondering if it would make more sense to have a generic map[string]interface{} that would allow users to configure any provider. We've had issues lately where teams want to change the provider version, update the source, or add providers not currently supported by fogg. This requires us to update the static list and push a release. Can you modify your provider updates PR to instead change fogg so that any provider can be specified with a generic configuration?

Multi-Module support

I'm not sure about this feature. Can you state your use-case? From my understanding about the origin of this project, fogg is good about setting up several projects in various different environments with only changes to the environment specific configuration. So prod/staging/dev all run the same HCL, with environment specific configuration. By supporting multiple modules per component, I feel like this breaks that convention a little. Prod might have different modules than staging, for instance. If you could state your use-case for this feature, I am open to understanding it better and maybe we can work on it from there.

[vincenthsh]

I'm not sure about this feature. Can you state your use-case?

in my use-case there are multiple modules per component.

I'm closing this PR for now, will try to make time to upstream some of the features you were interested in

unfortunately, the code base I adopted required multiple modules per component or it would require a lot of state mangling, so I will maintain my fork for now.

[vincenthsh]

I actually think https://github.com/chanzuckerberg/fogg/pull/497 should allow me to disable globalEnvs (and thus my code wasn't needed)