Sensitive values is missing

[serrovsky]

Hello guys

I'm facing an uncommon issue with Terraform that I think could be an edge case or a bug with tfe_output

We have several workspaces (one per env - test, staging, uat, and prod), but due to a costs issue, we have some more critical test resources under the staging workspace, like the Kubernetes cluster, key vault, container registry, etc...

But now, we have a data lake (basically a storage account with a bunch of blob containers) created under the test workspace but we need to create a secret on the test key vault with the storage account name and primary access key.

Once the data lake is created under the test workspace, but the test key vault is created under the staging workspace, we are trying to use outputs and tfe_outputs to use these values on different workspaces, so that, we added those as outputs on the test workspace

output "data_lake_storage_account_name" {
  value = module.data_lake.storage_account_name
}

output "data_lake_primary_access_key" {
  value = module.data_lake.primary_access_key
  sensitive = true
}

and on staging terraform files we set up tfe_outputs to get those values

data "tfe_outputs" "test_outputs" {
  organization = "my-org"
  workspace = "test"
}

and create the following azure key vault secrets (on staging):

resource "azurerm_key_vault_secret" "dev_data_lake_storage_account_name" {
  key_vault_id = module.akv-dev.id
  name         = "DataLakeAccountName"
  value        = data.tfe_outputs.test_outputs.values.data_lake_storage_account_name
}

resource "azurerm_key_vault_secret" "dev_data_lake_primary_access_key" {
  key_vault_id = module.akv-dev.id
  name         = "DataLakePrimaryAccessKey"
  value        = data.tfe_outputs.test_outputs.values.data_lake_primary_access_key
}

however, dev_data_lake_storage_account_name secret is created sucessfully, but dev_data_lake_primary_access_key secret is always giving this error:

╷
│ Error: Missing required argument
│ 
│   with azurerm_key_vault_secret.dev_data_lake_primary_access_key,
│   on secrets-dev.tf line 46, in resource "azurerm_key_vault_secret" "dev_data_lake_primary_access_key":
│   46:   value        = data.tfe_outputs.test_outputs.values.data_lake_primary_access_key
│ 
│ The argument "value" is required, but no definition was found.
╵

Not sure if this is because the output is marked as sensitive on the test workspace, but I can't find any documentation related to this. Can anyone help me with this problem? Am I the first person with this issue? I can't remove the sensitive flag from that output, because it's a access key and Terraform raises an error.

Thanks for your time and help guys

Terraform version

Terraform Cloud
1.1.7

Terraform Configuration Files

terraform {
  required_providers {
    azuread = {
      source = "hashicorp/azuread"
      version = "=1.5.1"
    }
    azurerm = {
      source = "hashicorp/azurerm"
      version         = "=2.80.0"
    }
......
    tfe = {
      source = "hashicorp/tfe"
      version = "0.28.1"
    }

  }
  required_version = ">= 1.0.0"
}

Debug Output

https://gist.github.com/luis-serra-ki/ddab4de4217674240f66689c0a4cffb8

Expected Behavior

resource "azurerm_key_vault_secret" "dev_data_lake_storage_account_name" created with success resource "azurerm_key_vault_secret" "dev_data_lake_primary_access_key" created with success

Actual Behavior

resource "azurerm_key_vault_secret" "dev_data_lake_storage_account_name" created with success resource "azurerm_key_vault_secret" "dev_data_lake_primary_access_key"error

[jmwri]

I am experiencing something similar but values are omitted and I'm not getting any errors.

My first workflow has the following outputs:

output "mysql_username" {
  value = var.mysql_username
}
output "mysql_password" {
  value     = var.mysql_password
  sensitive = true
}

The second workflow creates a kubernetes secret with the following data, however the mysql_password is not included the the created resource.

data = {
    mysql_username          = data.tfe_outputs.first-workspace.values.mysql_username
    mysql_password          = data.tfe_outputs.first-workspace.values.mysql_password
  }

[asaf-upstream]

Same issue here, all not sensitive values gets retrieved from the remote_state, but all sensitive values returns empty.

[asaf-upstream]

@serrovsky-pt @jmwri FYI, you may work with the terraform_remote_state data source instead of the tfe_outputs as a workaround, until the issue is solved:

data "terraform_remote_state" "infra_state" {
  backend = "remote"
  config = {
    organization = "xxx"
    workspaces = {
      name    = "yyy"
    }
   }
}
redis_password = data.terraform_remote_state.infra_state.outputs.redis_password

[qwerty1979bg]

Internally, the "tfe_outputs" resources uses the "Show Workspace" API call with the "outputs" include.

The documentation does not specifically state if this API call should also return the sensitive values, but in reality it definitely does not, leading to this issue.

[galvarado]

Same issue here, I needed to export the sensitive value as nonsensitive using https://www.terraform.io/language/functions/nonsensitive

[panasonixme]

Hi, any update on this bug ? It is typically the kind of issue that will prevent the adoption of this provider...

[spanktar]

I inadvertently filed the same bug under the main Terraform repo. Pasting it here for reference:

https://github.com/hashicorp/terraform/issues/31234

My workaround was to use nonsensitive() to get what I need working until this is fixed.

[tiagocborg]

I am also experiencing the same issue. The funny thing is it was working before, I created a whole setup using output. Now out of a sudden, I am having that issue with all the outputs

[marinoborges]

Got the same issue but solved this after sharing the state files with the remote state consumer as documented here https://www.terraform.io/docs/cloud/workspaces/state.html#accessing-state-from-other-workspaces. This worked both for tfe_outputs and terraform_remote_state.

[MXfive]

The sensitive outputs requires a code change here to catch if a secret is sensitive, then make an additional GET for to the state version outputs API here.

I am also experiencing the same issue. The funny thing is it was working before, I created a whole setup using output. Now out of a sudden, I am having that issue with all the outputs

It looks like something has broken on TFC side, we are using this extensively between a couple dozen workspaces, only some workspaces seem to be affected and started breaking at around mid last week also. Have reached out to support as there is no difference on the config we can manage (we Terraform our Terraform Cloud, so we know 🙂).

Just a few minutes ago we started to see a couple of our workspaces that have been broken for a few days start to work, but others not yet, so hopefully support will reply soon that a fix is being rolled out gradually.

[alexeyatbluescape]

@MXfive it's been 17 days, sounds like either TFC rest API needs to be adjusted or the provider like you mentioned. You said that it worked for some of your workspaces, but not others. Any changes since then? Any updates on the support ticket?

[MXfive]

@alexeyatbluescape Yes, through debugging with support it turned out that their development team made a breaking API that now requires consuming workspaces to be whitelisted as remote state consumers (or the source workspace sharing org wide to all workspaces).

They are apparently working on the docs and also a public release notes for the API. But I've not been following this as I've moved onto other tasks.

For the topic of this issue here where the provider does not support reading the sensitive outputs, I started working on a PR for this that I plan to test either this or next weekend. I'm not writing Go daily and it requires a bunch of refactoring to have a chance of being merged though. Will update here as I go.

[alexeyatbluescape]

Thanks for update @MXfive! Yep, workspace state sharing makes sense and I'm using it. I'm more concerned about sensitive output values. Thx for your input on that. I have also opened support ticket to put more urgency on the issue. I'm currently working around the issue using unsensitive function, but that is not sustainable and not secure.

[phill-lewis]

May or may not be relevant but I was having what I think is the same problem.

╷ │ Error: Unsupported attribute │ │ on main.tf line 15, in locals: │ 15: thing = data.tfe_outputs.outputter.values.anotherthing │ │ This object does not have an attribute named "anotherthing".

It turned out I needed to add the TFE_TOKEN environment variable to the workspace consuming the output as the tfe provider needs authentication to do work within terraform cloud. It is in the docs but I'd missed this because I'd been linked direct to the tfe_outputs page from wherever I'd been reading about using outputs in cloud. It was only when I was looking to use the tfe_provider to create a bootstrap workspace to build all other workspaces that I found the information.

Like I say it might not be the same issue everyone else is experiencing but it feels like it is.

The error message could have been more precise - pointing to an authentication issue would have pointed me in the right direction or maybe a change to the docs to add an information box on each page mentioning the need for auhentication.

Anyway hope this helps

Phill

[galvarado]

May or may not be relevant but I was having what I think is the same problem.

╷ │ Error: Unsupported attribute │ │ on main.tf line 15, in locals: │ 15: thing = data.tfe_outputs.outputter.values.anotherthing │ │ This object does not have an attribute named "anotherthing".

It turned out I needed to add the TFE_TOKEN environment variable to the workspace consuming the output as the tfe provider needs authentication to do work within terraform cloud. It is in the docs but I'd missed this because I'd been linked direct to the tfe_outputs page from wherever I'd been reading about using outputs in cloud. It was only when I was looking to use the tfe_provider to create a bootstrap workspace to build all other workspaces that I found the information.

Like I say it might not be the same issue everyone else is experiencing but it feels like it is.

The error message could have been more precise - pointing to an authentication issue would have pointed me in the right direction or maybe a change to the docs to add an information box on each page mentioning the need for auhentication.

Anyway hope this helps

Phill

Although it sounds related, the error occurs to me in TF cloud, so I can rule out that it is a token issue

[phill-lewis]

Although it sounds related, the error occurs to me in TF cloud, so I can rule out that it is a token issue

Hey Galvaro,

It was in cloud for me to. tfe_provider needs to have authentication to be able to do anything "workspace" related in cloud and accessing the outputs from one workspace from another in cloud is what I was trying to do and it was failing with the error - I created a token under my user and set it as TFE_TOKEN env variable in the consuming workspace and it worked. I've subsequently been told about the "share state" setting under workspace general and so now I use that.

Incidentally the tfe_workspace remote_state_consumer_ids suffers from the same problem tfe_variable_set workspace_ids suffered from which is it is difficult to manage the list programatically. In the case of tfe_variable_set workspace_ids they deprecated it and made tfe_workspace_variable_set which reverses the dependency making things a lot easier. I suspect at some point remote_state_consumer_ids will go the same way.

Anyway good luck getting things sorted hopefully you'll have a break thorugh soon :)

Cheers Phill

[mnazir23]

I have added the TFE_TOKEN as well yet the sensitive values are not retrieved.

This needs to be fixed before tfe_outputs can be used properly

[alexeyatbluescape]

should this be fixed via https://github.com/hashicorp/terraform-provider-tfe/pull/565 now? I need to try the latest version...

[tommoyang]

Although it sounds related, the error occurs to me in TF cloud, so I can rule out that it is a token issue

It's a similar problem, but this actually helped me find the issue! I had this working in one workspace but not another; It turns out the difference is that the workspace that did have the outputs working had Remote State Sharing in the workspace settings enabled for the target workspaces that needed it. I did the same for the workspace that wasn't working, and it started working for that one too.

(Well, actually it was enabled for all workspaces in the org, but it defaults to targeted workspaces with an empty list)

[MXfive]

Although it sounds related, the error occurs to me in TF cloud, so I can rule out that it is a token issue

It's a similar problem, but this actually helped me find the issue! I had this working in one workspace but not another; It turns out the difference is that the workspace that did have the outputs working had Remote State Sharing in the workspace settings enabled for the target workspaces that needed it. I did the same for the workspace that wasn't working, and it started working for that one too.

(Well, actually it was enabled for all workspaces in the org, but it defaults to targeted workspaces with an empty list)

Correct, there are two completely seperate things:

1. A bug in this provider that was fixed in #565
2. A breaking change made on the TFC API which now requires the source workspace to allow state sharing with the consumer workspace

This ticket is only around no 1. and so should be close now IMO.

[brandonc]

I believe this issue was addressed by resolving https://github.com/hashicorp/terraform-provider-tfe/issues/557