kafka-connect-lambda

kafka-connect-lambda copied to clipboard

Hi, I have running a big Kafka Connect cluster using Sink Lambda connector, we have 450 connectors and 2700 tasks and 12 workers (22GB reserved to xmx) and we are...

dinegri

Batch max size question

4

We are attempting to reduce the number of messages being processed by each lambda invocation. We have batching enabled but it doesn't appear that you can configure the message count....

Amertz08

Maybe this behaviour should be configurable, but this will at least not swallow any function execution error that occurs. Based on [aws doc](https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html#API_Invoke_ResponseSyntax) Fixes https://github.com/Nordstrom/kafka-connect-lambda/issues/35 Sample log output upon function...

RickardCardell

Sync mode should stop task when Lambda function fails

10

Thanks for a nice plugin! When using sync mode and the invocation succeeds the response is seen as [success](https://github.com/Nordstrom/kafka-connect-lambda/blob/master/src/main/java/com/nordstrom/kafka/connect/lambda/LambdaSinkTask.java#L210) even though the Lambda execution failed. As long as the invocation...

Cardell

Response back to kafka

2

I was playing around with using the response from the lambda to put it back on a different topic. Have you had this requirement and did you solve it by...

gabrielsson

Document advantages over llofberg library

1

Greetings, great to see some activity in this area. I was wondering if it would be possible to document the advantages or differences of this project to the https://github.com/llofberg/kafka-connect-aws-lambda (which...

matt-bates-hypr