Amazon Kinesis Support Enhanced Fan-Out

[pauldheinrichs]

Description

Feel free to close this if there is a better place to put this or if I'm incorrect. But I feel like there are improvements that could be made to the kinesis ingestion to resolve the main "known issue" under the kinesis druid docs.

Before you deploy the Kinesis extension to production, consider the following known issues: Avoid implementing more than one Kinesis supervisor that reads from the same Kinesis stream for ingestion. Kinesis has a per-shard read throughput limit and having multiple supervisors on the same stream can reduce available read throughput for an individual supervisor's tasks.

What does Enhanced fan out do?

A consumer that uses enhanced fan-out doesn't have to contend with other consumers who are receiving data from the stream

Source Docs:

• https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/
• V1 docs: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesis.html#putRecords-com.amazonaws.services.kinesis.model.PutRecordsRequest-

https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

---

I have not dug into the kinesis consuming code enough to say what the lift would be to support this, but thought posting this for visibility might be worth while? :shrug:

[yurmix]

This will require solving https://github.com/apache/druid/issues/16903

[github-actions[bot]]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[gianm]

Still seems useful. I believe it still requires the upgrade to SDK 2 described in #16903.