amazon-kinesis-producer
amazon-kinesis-producer copied to clipboard
awslabs
Will KPL support writing to kinesis firehose?
We do plan to update the KPL to support Firehose. Unfortunately we do not have a date yet.
I was told by my AWS rep on July 2, 2016 that KPL does not support firehose and there is not intention for Amazon engineers to do that...they are leaving it up to the community to write that piece of code...
It seems there is interest in adding support for Firehose to the KPL, and it's something we're investigating. We are prioritizing the adding the support with other customer requests.
Other please comment, or add reactions to assist us in prioritizing the issue.
Would be great if it was supported for Firehose, to save me re-implementing all the logic myself.
If there was a Lambda Blueprint for transforming KPL records in a Firehose Stream that would be helpful.
Is this about writing directly to Firehose, without going through a data stream? Instead of "KPL → Kinesis Data Stream → Kinesis Data Firehose" you have "KPL → Kinesis Data Firehose"?
It seems a bit odd to do that, because with a direct PUT you still have the 5000 limit, where if you used the stream you'd not be subject to that.
https://docs.aws.amazon.com/firehose/latest/dev/limits.html
"When Kinesis Data Streams is configured as the data source, this limit doesn't apply, and Kinesis Data Firehose scales up and down with no limit."
EDIT: Ah, I see the pricing difference between Kinesis Data Streams and Firehose now...
Probably the best thing to do is use https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md and https://github.com/awslabs/kinesis-aggregation/tree/master/java/KinesisAggregator with your own streaming implementation.
Would be great if it was supported for Firehose, to save me re-implementing all the logic myself.
