aws_secretsmanager_secret_version timing issue

[syst0m]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.28

• provider.aws v2.70.0
• provider.random v2.3.0

Affected Resource(s)

• aws_secretsmanager_secret
• aws_secretsmanager_secret_version

Terraform Configuration Files

resource "aws_secretsmanager_secret" "ume_redshift_password" {
  name = "ume-redshift-password"
}

resource "aws_secretsmanager_secret_version" "ume_redshift_password" {
  secret_id     = aws_secretsmanager_secret.ume_redshift_password.id
  secret_string = random_password.ume_redshift_password.result
}

resource "random_password" "ume_redshift_password" {
  length      = 20
  special     = false
  min_lower   = 1
  min_upper   = 1
  min_numeric = 1
}

data "aws_secretsmanager_secret_version" "ume_redshift_password" {
  secret_id = aws_secretsmanager_secret.ume_redshift_password.id
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

The random_password resource generates a random password, stores in a secretsmanager secret. The data source retrieves the secret_string attribute, and can be used for interpolation in other places of the config (not shown here).

Actual Behavior

The data source fails to retrieve the AWSCURRENT staging version of the secret.

Error: Secrets Manager Secret "arn:aws:secretsmanager:XXXX:secret:ume-redshift-password-uclIsh" Version "AWSCURRENT" not found

The error is gone after applying a 2nd time, and the apply finishes successfully.

Steps to Reproduce

1. terraform apply

Important Factoids

N/A

References

N/A

[Oliniusz]

I have the similar issue with the following code:

resource "random_password" "rds_admin" {
  length  = 16
  special = false
}

resource "aws_secretsmanager_secret" "rds_admin" {
  provider                = aws.eu-west-1
  name                    = "db_admin"
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "rds_admin" {
  provider      = aws.eu-west-1
  secret_id     = "db_admin"
  secret_string = random_password.rds_admin.result
}

After the first run of terraform apply I get the error:

Error: error putting Secrets Manager Secret value: ResourceNotFoundException: Secrets Manager can't find the specified secret.

The second terraform apply runs as expected. I guess I could try to put some sleep time as a workaround but I haven't tried it yet.

provider.aws: version = "~> 3.10"

[gneveu]

Hello,

Does it help if you replace secret_id = "db_admin" in the aws_secretsmanager_secret_version by secret_id = aws_secretsmanager_secret.rds_admin.id ?

[Oliniusz]

Hi,

I see what you mean.

For now I've used depends on instead and it also seems to work:

resource "aws_secretsmanager_secret_version" "rds_dbadmin" {
  provider      = aws.eu-west-1
  secret_id     = "db_dbadmin"
  secret_string = random_password.rds_dbadmin.result
  depends_on    = [aws_secretsmanager_secret.rds_dbadmin]
}

But I think your secret_id = aws_secretsmanager_secret.rds_admin.id is cleaner and more elegant - I'm changing my code now.

[justinretzolk]

Hi @syst0m 👋 Thank you for reporting this. Given the comments above and that there's been a few Terraform and AWS provider releases between when you filed this and now, can you confirm whether you're still experiencing this?

[digihunch]

I am still experiencing this issue at random.

{"@level":"info","@message":"module.secretmanager.data.aws_secretsmanager_secret_version.creds: Still refreshing... [1m20s elapsed]","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:00.850905Z","hook":{"resource":{"addr":"module.secretmanager.data.aws_secretsmanager_secret_version.creds","module":"module.secretmanager","resource":"data.aws_secretsmanager_secret_version.creds","implied_provider":"aws","resource_type":"aws_secretsmanager_secret_version","resource_name":"creds","resource_key":null},"action":"read","elapsed_seconds":80},"type":"apply_progress"}
{"@level":"info","@message":"module.secretmanager.aws_vpc_endpoint.secmgr: Still creating... [1m20s elapsed]","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:07.160941Z","hook":{"resource":{"addr":"module.secretmanager.aws_vpc_endpoint.secmgr","module":"module.secretmanager","resource":"aws_vpc_endpoint.secmgr","implied_provider":"aws","resource_type":"aws_vpc_endpoint","resource_name":"secmgr","resource_key":null},"action":"create","elapsed_seconds":80},"type":"apply_progress"}
{"@level":"info","@message":"module.secretmanager.aws_vpc_endpoint.secmgr: Creation complete after 1m22s [id=vpce-096cb4d9b732bab91]","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:09.151525Z","hook":{"resource":{"addr":"module.secretmanager.aws_vpc_endpoint.secmgr","module":"module.secretmanager","resource":"aws_vpc_endpoint.secmgr","implied_provider":"aws","resource_type":"aws_vpc_endpoint","resource_name":"secmgr","resource_key":null},"action":"create","id_key":"id","id_value":"vpce-096cb4d9b732bab91","elapsed_seconds":82},"type":"apply_complete"}
{"@level":"error","@message":"Error: Secrets Manager Secret \"arn:aws:secretsmanager:us-east-1:111222333444:secret:hopeful-foxDatabaseCreds-X80WfS\" Version \"AWSCURRENT\" not found","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:09.349632Z","diagnostic":{"severity":"error","summary":"Secrets Manager Secret \"arn:aws:secretsmanager:us-east-1:111222333444:secret:hopeful-foxDatabaseCreds-X80WfS\" Version \"AWSCURRENT\" not found","detail":"","address":"module.secretmanager.data.aws_secretsmanager_secret_version.creds","range":{"filename":"modules/secmgr/data.tf","start":{"line":19,"column":50,"byte":387},"end":{"line":19,"column":51,"byte":388}},"snippet":{"context":"data \"aws_secretsmanager_secret_version\" \"creds\"","code":"data \"aws_secretsmanager_secret_version\" \"creds\" {","start_line":19,"highlight_start_offset":49,"highlight_end_offset":50,"values":[]}},"type":"diagnostic"}

After cleaning up and retry, then everything works again.

[justinretzolk]

Hey @digihunch 👋 Can you confirm what version of Terraform and the AWS Provider you're using?

[digihunch]

I was on terraform v1.0.11, with AWS provider 3.12.0 I'll try the latest version of both

[girvenj]

I am seeing the original issue as well:

Terraform v1.2.4

AWS provider: 4.24.0

[justinretzolk]

For those still experiencing this issue: The original issue here seems to have been a result of the data.aws_secretsmanager_secret_version being read prior to the aws_secretsmanager_secret_version resource creation.

This can likely be resolved by adding a depends_on block to set up an explicit dependency on the aws_secretsmanager_secret_version resource. Alternatively, data.aws_secretsmanager_secret_version's secret_id could be set to aws_secretsmanager_secret_version.<name>.secret_id to create an implicit dependency.

If you're still experiencing this issue, can you test this to confirm that it resolves your issue?

[jamiegosling]

This still seems to be present in Terraform 1.2.7 with AWS provider version 4.43.0. I have an code block like the following:

resource "random_string" "dbpass" {
  length  = 22
  upper   = true
  lower   = true
  numeric  = true
  special = false
}

locals {
  masterpasswd = random_string.dbpass.result
}

resource "aws_secretsmanager_secret" "password" {
  name = "name"
  force_overwrite_replica_secret = true
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "password" {
  secret_id     = aws_secretsmanager_secret.masterpasswd.id
  secret_string = local.masterpasswd

  depends_on = [
    aws_secretsmanager_secret.masterpasswd
  ]
}

which fails on the first apply

Error: Secrets Manager Secret "arn:aws:secretsmanager:eu-west-2:xxx:secret:xxxx-zIQt0L" Version "AWSCURRENT" not found

and works on the subsequent one.

[mims92]

Still occurring with Terraform 1.3.9 and AWS provider 4.55

[uridium]

Same with:

Terraform v1.4.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.65.0

[justinretzolk]

Given that this was initially reported on a much older version of the AWS provider, can someone who has run into this issue more recently provider a sample Terraform configuration for us to review?

[YusDyr]

I had this error with that configuration:

test.tfvars

target_endpoint = {
  "dst-all" = {
    engine_name = "docdb"
    port        = 27017
    server_name = "mongodb+srv://latam-tech-dev.btrq7.mongodb.net"
    secret_arn  = "arn:aws:secretsmanager:us-east-1:***:secret:/latam-tech/mongo-atlas/env/dev/credentials/admin/latam-tech-dev-jhRNCn"
    ssl_mode    = "none"
  }
}

main.tf

locals {
  targets_secret_arn = {
    for k, v in var.target_endpoint : k => v.secret_arn
    if contains(keys(v), "secret_arn")
  }
  targets_secret = { for k, v in data.aws_secretsmanager_secret_version.targets_endpoint : k => v.secret_string }
}

variable "target_endpoint" {
  description = "Map of objects that define target endpoint to be created, refer to https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dms_endpoint"
  type        = any
}

data "aws_secretsmanager_secret_version" "targets_endpoint" {
  for_each  = local.targets_secret_arn
  secret_id = each.value
}

Result: terraform apply -var-file=test.tfvars

data.aws_secretsmanager_secret_version.targets_endpoint["dst-all"]: Reading...
╷
│ Error: Secrets Manager Secret "arn:aws:secretsmanager:us-east-1:***:secret:/latam-tech/mongo-atlas/env/dev/credentials/admin/latam-tech-dev-jhRNCn" Version "AWSCURRENT" not found
│ 
│   with data.aws_secretsmanager_secret_version.targets_endpoint["dst-all"],
│   on 1.tf line 14, in data "aws_secretsmanager_secret_version" "targets_endpoint":
│   14: data "aws_secretsmanager_secret_version" "targets_endpoint" {
│ 
╵

Versions:

$ tf version
Terraform v1.2.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v5.15.0

The same story with terraform version 1.4.5

Remark: I replaced aws account id with "***"

[YusDyr]

I found the root cause. AWS_REGION was set to us-west-2 in my case, while used secret arn was for us-east-1!