Support for multistream processing

[stevebakh]

KCL 2 supports multi-stream processing. Developers can configure a list of streams, and the KCL uses a single lease table. Is there any appetite to support this?

https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html#shared-throughput-kcl-multistream

[semenodm]

Hi Steven. Well, you can potentially do this. there are 2 factory methods for kinesis fs2 steams

• Default
• One that allows scheduler factory customization

The default one is most easy to use, you pass all predefined required clients, and you are ready to stream. However, the ConfigBuilder is hidden from you, and you cannot provide the multi-steam config as the library user. The second one was created mostly because we needed to test our implementation, and at the time we started this library, there were scarce resources for mocking kinesis SDK, especially to mimic such cases as shard ends, etc. So you can potentially use the second factory method and make your customization. However, we need to spend more time figuring out how to make it more user-friendly.

[semenodm]

@stevebakh hi Steve, here is new proposal on how to build kinesis streaming #1030

[stevebakh]

Awesome, thanks Dmytro, and apologies for the radio silence! Will take a closer look at the new solution - we're currently just creating two consumers, each with their own lease table. Looking forward to seeing how we can integrate the above.