Firehose describe_delivery_stream does not give expected response to stream created with S3 backup

[randomcoder]

I am creating a firehose delivery stream, with either S3 or Redshift delivery options and trying to set the S3 backup option. The creation request (converted to Json from the boto args)

{
  "DeliveryStreamName": "simple-firehose-to-s3",
  "Tags": [
    {
      "Key": "ai_managed",
      "Value": "True"
    }
  ],
  "DeliveryStreamType": "DirectPut",
  "RedshiftDestinationConfiguration": {
    "RoleARN": "arn:aws:iam:eu-west-1:123456789012:firehose_role",
    "ClusterJDBCURL": "jdbc:redshift://**********.eu-west-1.redshift.amazonaws.com:5439/database",
    "CopyCommand": {
      "DataTableName": "db.test_table",
      "DataTableColumns": "column1",
      "CopyOptions": "TEST COPY COMMAND"
    },
    "Username": "********",
    "Password": "dummy-password",
    "RetryOptions": {
      "DurationInSeconds": 3600
    },
    "S3Configuration": {
      "RoleARN": "arn:aws:iam:eu-west-1:123456789012:firehose_role",
      "BucketARN": "arn:aws:s3:::replication-data",
      "Prefix": "dummy/export_data/",
      "ErrorOutputPrefix": "dummy/error_data/",
      "BufferingHints": {
        "SizeInMBs": 3,
        "IntervalInSeconds": 300
      },
      "CompressionFormat": "UNCOMPRESSED",
      "EncryptionConfiguration": {
        "NoEncryptionConfig": "NoEncryption"
      }
    },
    "ProcessingConfiguration": {
      "Enabled": false
    },
    "CloudWatchLoggingOptions": {
      "Enabled": true,
      "LogGroupName": "aws/kinesisfirehose/simple-firehose-to-s3",
      "LogStreamName": "DestinationDelivery"
    },
    "S3BackupMode": "Enabled",
    "S3BackupConfiguration": {
      "RoleARN": "arn:aws:iam:eu-west-1:123456789012:firehose_role",
      "BucketARN": "arn:aws:s3:::test-bucket",
      "Prefix": "dummy/source_data/",
      "ErrorOutputPrefix": "dummy/source_error_data/",
      "BufferingHints": {
        "SizeInMBs": 3,
        "IntervalInSeconds": 300
      },
      "CompressionFormat": "UNCOMPRESSED",
      "EncryptionConfiguration": {
        "NoEncryptionConfig": "NoEncryption"
      },
      "CloudWatchLoggingOptions": {
        "Enabled": true,
        "LogGroupName": "aws/kinesisfirehose/simple-firehose-to-s3",
        "LogStreamName": "BackupDelivery"
      }
    }
  }
}

The creation request is successful however when I get the stream description with

client = boto3.client('firehose', 'eu-west-1')
description = client-describe_delivery_stream(DeliveryStreamName='simple-firehose-to-s3')

the response I get is

{
  "DeliveryStreamDescription": {
    "DeliveryStreamName": "simple-firehose-to-s3",
    "DeliveryStreamARN": "arn:aws:firehose:eu-west-1:123456789012:deliverystream/simple-firehose-to-s3",
    "DeliveryStreamStatus": "ACTIVE",
    "DeliveryStreamType": "DirectPut",
    "VersionId": "1",
    "CreateTimestamp": "2022-06-23 10:18:53.777672+00:00",
    "LastUpdateTimestamp": "2022-06-23 10:18:53.777685+00:00",
    "Destinations": [
      {
        "DestinationId": "destinationId-000000000001",
        "RedshiftDestinationDescription": {
          "RoleARN": "arn:aws:iam:eu-west-1:123456789012:firehose_role",
          "ClusterJDBCURL": "jdbc:redshift://**********.eu-west-1.redshift.amazonaws.com:5439/database",
          "CopyCommand": {
            "DataTableName": "db.test_table",
            "DataTableColumns": "column1",
            "CopyOptions": "TEST COPY COMMAND"
          },
          "Username": "********",
          "RetryOptions": {
            "DurationInSeconds": 3600
          },
          "S3DestinationDescription": {
            "RoleARN": "arn:aws:iam:eu-west-1:123456789012:firehose_role",
            "BucketARN": "arn:aws:s3:::replication-data",
            "Prefix": "dummy/export_data/",
            "ErrorOutputPrefix": "dummy/error_data/",
            "BufferingHints": {
              "SizeInMBs": 3,
              "IntervalInSeconds": 300
            },
            "CompressionFormat": "UNCOMPRESSED",
            "EncryptionConfiguration": {
              "NoEncryptionConfig": "NoEncryption"
            }
          },
          "ProcessingConfiguration": {
            "Enabled": false
          },
          "S3BackupMode": "Enabled",
          "CloudWatchLoggingOptions": {
            "Enabled": true,
            "LogGroupName": "aws/kinesisfirehose/simple-firehose-to-s3",
            "LogStreamName": "DestinationDelivery"
          }
        }
      }
    ],
    "HasMoreDestinations": false
  }
}

which has both RedshiftDestinationDescription and S3DestinationDescription.

A comparable response from AWS is

{
  "DeliveryStreamDescription": {
    "DeliveryStreamName": "simple-firehose",
    "DeliveryStreamARN": "arn:aws:firehose:eu-west-1:463813521958:deliverystream/tbone-panda-article",
    "DeliveryStreamStatus": "ACTIVE",
    "DeliveryStreamEncryptionConfiguration": {
      "Status": "DISABLED"
    },
    "DeliveryStreamType": "DirectPut",
    "VersionId": "1",
    "CreateTimestamp": "2022-06-14T09:41:57.855000+01:00",
    "Destinations": [
      {
        "DestinationId": "destinationId-000000000001",
        "RedshiftDestinationDescription": {
          "RoleARN": "arn:aws:iam::123456789012:role/service-role/KinesisFirehoseServiceRole",
          "ClusterJDBCURL": "jdbc:redshift://********.eu-west-1.redshift.amazonaws.com:5439/database",
          "CopyCommand": {
            "DataTableName": "schema.table",
            "DataTableColumns": "*******,********,********",
            "CopyOptions": "FORMAT AS CSV timeformat 'auto' emptyasnull blanksasnull"
          },
          "Username": "*******",
          "RetryOptions": {
            "DurationInSeconds": 0
          },
          "S3DestinationDescription": {
            "RoleARN": "arn:aws:iam::123456789012:role/service-role/KinesisFirehoseServiceRole",
            "BucketARN": "arn:aws:s3:::backup-data-test",
            "Prefix": "dummy/export-data/",
            "BufferingHints": {
              "SizeInMBs": 1,
              "IntervalInSeconds": 60
            },
            "CompressionFormat": "UNCOMPRESSED",
            "EncryptionConfiguration": {
              "NoEncryptionConfig": "NoEncryption"
            },
            "CloudWatchLoggingOptions": {
              "Enabled": true,
              "LogGroupName": "aws/kinesisfirehose/simple-firehose",
              "LogStreamName": "DestinationDelivery"
            }
          },
          "ProcessingConfiguration": {
            "Enabled": false
          },
          "S3BackupMode": "Enabled",
          "S3BackupDescription": {
            "RoleARN": "arn:aws:iam::123456789012:role/service-role/KinesisFirehoseServiceRole",
            "BucketARN": "arn:aws:s3:::backup-data-test",
            "Prefix": "dummy/source_data/",
            "BufferingHints": {
              "SizeInMBs": 1,
              "IntervalInSeconds": 60
            },
            "CompressionFormat": "UNCOMPRESSED",
            "EncryptionConfiguration": {
              "NoEncryptionConfig": "NoEncryption"
            },
            "CloudWatchLoggingOptions": {
              "Enabled": true,
              "LogGroupName": "aws/kinesisfirehose/simple-firehose",
              "LogStreamName": "BackupDelivery"
            }
          },
          "CloudWatchLoggingOptions": {
            "Enabled": true,
            "LogGroupName": "aws/kinesisfirehose/simple-firehose",
            "LogStreamName": "DestinationDelivery"
          }
        }
      }
    ],
    "HasMoreDestinations": false
  }
}

[bblommers]

Hi @randomcoder, thanks for raising this - marking it as an enhancement.

If this is something you want to implement yourself, PR's are always welcome!

[kbalk]

Processing of the S3BackupMode argument was never implemented. Not all of the destinations are implemented either. A list of the unimplemented features are at the top of the file: https://github.com/spulec/moto/blob/master/moto/firehose/models.py