Firehose delivery stream arn consumers such as aws_cloudwatch_log_subscription_filter should wait for the stream to become ACTIVE before attempting to use it

[MartiUK]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

At the moment if a subscription filter destination_arn is set to a firehose delivery stream arn, it will attempt to create as soon as the firehose delivery stream is created, rather than when the stream is in an ACTIVE state, this means that any runs that require both a firehose delivery stream and a subscription filter to be created will fail as the subscription filters will time out waiting for the stream to become ACTIVE.

The resulting error is below: Error creating Cloudwatch log subscription filter: InvalidParameterException: Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state.

New or Affected Resource(s)

• aws_cloudwatch_log_subscription_filter
• aws_kinesis_firehose_delivery_stream

Potential Terraform Configuration

resource aws_kinesis_firehose_delivery_stream metrics {
  name        = "metrics-${terraform.workspace}"
  destination = "extended_s3"

  extended_s3_configuration {
    bucket_arn = aws_s3_bucket.service_logs.arn
    role_arn   = aws_iam_role.firehose.arn

    processing_configuration {
      enabled = true
      processors {
        type = "Lambda"

        parameters {
          parameter_name  = "LambdaArn"
          parameter_value = aws_lambda_function.metrics_collector.qualified_arn
        }
      }
    }
  }
}

resource aws_cloudwatch_log_subscription_filter metrics {
  log_group_name  = aws_cloudwatch_log_group.logs.name
  name            = "${var.service.name}-${terraform.workspace}-metrics"
  destination_arn = aws_kinesis_firehose_delivery_stream.metrics.arn
  role_arn        = var.metrics_fh_role
  filter_pattern  = ""
}

References

• https://docs.aws.amazon.com/firehose/latest/APIReference/API_CreateDeliveryStream.html

  This is an asynchronous operation that immediately returns. The initial status of the delivery stream is CREATING. After the delivery stream is created, its status is ACTIVE and it now accepts data. If the delivery stream creation fails, the status transitions to CREATING_FAILED. Attempts to send data to a delivery stream that is not in the ACTIVE state cause an exception.

[anGie44]

Hi @MartiUK, thank you for raising this issue. To investigate this behavior you are facing, do you mind providing the provider version you are working with as well as any debug/output logging you may have? currently we catch that error you are seeing and retry resource creation for 5 minutes as seen here https://github.com/hashicorp/terraform-provider-aws/blob/d18d9c04cf6c18a49162435170815cea81c78cd3/aws/resource_aws_cloudwatch_log_subscription_filter.go#L71 before timing out so I suspect that limit could be part of the issue if the delivery stream takes the maximum time allowed to create i.e. 20min as defined in the resource code.

[MartiUK]

@anGie44 This is with 3.23.0. Funnily enough when tainting the firehose delivery stream and reapplying, there is no issue, it's only on completely brand new streams.

[bleachbyte]

Any information about when this will be updated?

[dschro-1993]

Any updates?

[gregtws]

This impacted us as well. Waiting a bit and re-running TF lets us finish the run, but its a pretty poor experience.