Feature: Partial environment variables

[a-schot]

Is your feature request related to a problem? Please describe.

When provisioning environment variables through the resource dbtcloud_environment_variable, the environment already has to be deployed in order to give a value to the environment variable. When deploying dbt environments using separate state files this causes an issue where you can only really deploy the environment variables during the deployment of the last environment (eg. PROD)

Describe the solution you'd like

A dbtcloud_partial_environment_variable resource, akin to the dbtcloud_partial_license_map resource.

Describe alternatives you've considered

We are currently using count = var.environment == local.last_environment ? 1 : 0, but want a more flexible solution

Additional context

[b-per]

Thanks for raising this.

I might try to have a look in the next few days/weeks but our APIs for env variables is not the best to interact with 😄

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 90 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[a-schot]

Hi @b-per, have you had any chance to look at this yet?

[b-per]

Hi @a-schot . No, I haven't had the time. There are a few other new resources/datasources that we want to add as well and I don't know when we will have the time to tackle this one.

[WobblyRobbly]

@a-schot I just want to clarify what you are proposing as we have a similar requirement.

As part of our CI pipeline we don't want to deploy changes to production until that part of the CI pipeline executes as a separate job. If we have 3 environments

dev, test and prod

Are you proposing that the dbtcloud_partial_environment_variable is run once for dev and test and then in a separate job with its own state file for prod , applying all prod values ?

Should the provider be able to apply single values at the project/environment/variable level instead ?

[gunnava]

Hi @b-per , We received this request from another customer.

Currently the dbt cloud terraform provider requires you to create the values for all environments at the same time. Our approach to CI/CD means we want to only create the production values when a production CI/CD job is executed - we need to have gone through our change management process. Currently if we are on a feature branch it is possible to use the existing environment_variable resource to update the production variable values and thus affect the existing production executions

[b-per]

This has been raised a few times but I don't know if/when we will be able to accomodate the partial resource for env var.

In the meantime, I wanted to share a workaround.

If today, people us the use env_var("DBT_ENV_MY_VAR") in their dbt project and want to be able to provide the value for DBT_ENV_MY_VAR as part of different dbt project, an alternative could be to update the dbt code to do the following instead: env_var("DBT_ENV_MY_VAR_PROD", env_var("DBT_ENV_MY_VAR"))

This means that we would look in the current env if DBT_ENV_MY_VAR_PROD is set.

• if it is set, we get its value
• if it is not, it means we are not in Prod and in that case we get the value of DBT_ENV_MY_VAR.

That way instead of updating DBT_ENV_MY_VAR, the Prod TF project can just set the new env var DBT_ENV_MY_VAR_PROD with the relevant value for Prod.

This is not perfect and requires changing the dbt code, but it would work with the way the current env var resource works.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 90 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[WobblyRobbly]

Still very much interested in this

[b-per]

Hi! I am getting less involved in the development of the provider as another team is taking over, but I believe that they added this feature in this PR that will be part of v 1.0.0 that should be released soon.

[victorasu]

As @b-per said, Partial Environment Variables will be released as part of the upcoming 1.0.0 version. In the meantime, since we're focused on the Open-Source factor, anyone interested can share their thoughts on the code we've pushed.

~We'll be keeping the issue open for a while longer after the release, to analyze the impact of the feature on your use-cases.~ On further consideration, we'll be tracking this internally. Should any issues come up with the implementation, we'll be expecting Bug submissions.