How to handle backpressure whe using a KCL-KPL application written using cloud-stream-kinesis-binder

[siddharthjain210]

In what version(s) of Spring Cloud Stream Binder for AWS Kinesis are you seeing this issue? 3.0.0

Describe the bug I have an application written using spring-cloud-stream, that connects 2 different kinesis streams by relaying the records from source stream to the destination streams. The streams can be in different aws accounts too. The setup is to define a kinesis bridge between 2 different systems / accounts.

the application uses a KCL consumer to read records from a stream in source account and publishes messages to a stream in the destination aws account using KPL.

In heavy load, we are losing records when publishing to KPL, the errors and exceptions are swallowed internally. The KCL is working perfectly fine.

How do we configure backpressure when using spring-cloud-stream-kinesis-binder as referred to the below links

https://aws.amazon.com/blogs/big-data/implementing-efficient-and-reliable-producers-with-the-amazon-kinesis-producer-library/ https://github.com/awslabs/amazon-kinesis-producer?tab=readme-ov-file#back-pressure

To Reproduce Setup a KCL-KPL enables application, produce a very heavy load

Expected behavior under extreme load, there shouldnt be any record loss when using KPL. There should be capability to handle backpressue, even if PutRecords Aggregation is enabled.

[siddharthjain210]

Digging further deeper, this requires a change in the KPLMessageHandler.handleUserRecord method in spring-integration-aws library.

[artembilan]

I think this one would be a natural evolution after https://github.com/spring-projects/spring-integration-aws/issues/249. So, we would expose respective back-pressure configuration properties and handle them in the binder for KPL binding configuration. Thanks